Conditional-reset, multi-bit read-out image sensor

ABSTRACT

An image sensor architecture with multi-bit sampling is implemented within an image sensor system. A pixel signal produced in response to light incident upon a photosensitive element is converted to a multiple-bit digital value representative of the pixel signal. If the pixel signal exceeds a sampling threshold, the photosensitive element is reset. During an image capture period, digital values associated with pixel signals that exceed a sampling threshold are accumulated into image data.

TECHNICAL FIELD

The present disclosure relates to the field of electronic image sensors,and more specifically to a sampling architecture for use in such imagesensors.

BACKGROUND

Digital image sensors, such as CMOS or CCD sensors, include a pluralityof photosensitive elements (“photosensors”) each configured to convertphotons incident upon the photosensors (“captured light”) into electriccharge. The electric charge can then be converted into image datarepresenting the light captured by each photosensor. The image dataincludes a digital representation of the captured light, and may bemanipulated or processed to produce a digital image capable of displayon a viewing device. Image sensors are implemented in integratedcircuits (“ICs”) with a physical surface that may be divided into aplurality of pixel regions (for instance, one or more photosensors andattendant control circuitry) configured to convert light into anelectrical signal (charge, voltage, current, etc.). For convenience,pixel regions within an image sensor may also be referred to as imagepixels (“IPs”) and the aggregate of the pixel regions or image pixelswill be referred to as the image sensor region. An image sensor ICtypically will also include areas outside of the image sensor region,for example certain types of control, sampling, or interface circuitry.Most CMOS image sensors contain A/D (analog-to-digital) circuitry toconvert pixel electrical signals into digital image data. The A/Dcircuitry can be one or more ADCs (analog-to-digital converters) locatedwithin or at the periphery of the image sensor region.

BRIEF DESCRIPTION OF THE DRAWINGS

The various embodiments disclosed herein are illustrated by way ofexample, and not by way of limitation, in the figures of theaccompanying drawings and in which like reference numerals refer tosimilar elements and in which:

FIG. 1 illustrates a cross-section of a portion of an image sensor,according to one embodiment;

FIG. 2 illustrates partial array circuitry of an analog pixel imagesensor with multiple pixel signal thresholds, according to oneembodiment useful, e.g., in the layout of FIG. 1;

FIG. 3 illustrates an example image sensor read circuit configured toconvert a pixel signal into a multi-bit digital conversion, according toone embodiment useful, e.g., with the embodiments of FIGS. 1 and 2;

FIG. 4 illustrates an example circuit block diagram embodiment of animage sensor system with a multi-bit architecture, according to oneembodiment using, e.g., the cross-section of FIG. 1 and the circuitry ofFIGS. 2 and 3;

FIG. 5 illustrates another example circuit block diagram of an imagesensor system architecture with read circuit arrays located peripherallyto an IP array, according to one embodiment using, e.g., thecross-section of FIG. 1 and the circuitry of FIGS. 2 and 3;

FIG. 6a illustrates a top view of a pixel array IC in an exampletwo-layer image sensor system architecture alternative to FIGS. 4 and 5,according to one embodiment using, e.g., the array circuitry of FIG. 2;

FIG. 6b illustrates a top view of a preprocessor IC in an exampletwo-layer image sensor system architecture alternative to FIGS. 4 and 5,according to one embodiment using, e.g., the read circuitry of FIG. 3;

FIG. 6c illustrates a partial cross section of the pixel array IC ofFIG. 6a and the preprocessor IC of FIG. 6b in an example two-layer imagesensor system architecture, according to one embodiment;

FIG. 7 illustrates the operation of an image sensor read circuit, suchas the read circuit of FIG. 3, according to one embodiment;

FIG. 8 illustrates data flow in an image capture system, according toone embodiment useful with the systems described herein;

FIG. 9 illustrates various temporal sampling policies for use by animage sensor read circuit, such as the read circuit of FIG. 3, accordingto one embodiment;

FIG. 10 illustrates an embodiment of a modified 4-transistor pixel inwhich a non-destructive over-threshold detection operation is executedto enable conditional-reset operation in conjunction with correlateddouble sampling;

FIG. 11 is a timing diagram illustrating an exemplary pixel cycle withinthe progressive read-out pixel of FIG. 10;

FIGS. 12 and 13 illustrate exemplary electrostatic potential diagramsfor the photodiode, transfer gate and floating diffusion of FIG. 10below their corresponding schematic cross-section diagrams;

FIG. 14 illustrates an embodiment of an image sensor 300 having aprogressive-readout pixel array;

FIG. 15 illustrates an alternative conditional-reset pixel embodimenthaving a transfer gate disposed between photosensitive element andgate-controlled sense node to enable correlated double-sampling;

FIG. 16 illustrates exemplary operational phases within a pixel cycle ofthe conditional-reset pixel of FIG. 15;

FIG. 17 is a timing diagram corresponding to FIG. 16 showing exemplarycontrol signal states generated during each phase of operation withinthe conditional-reset pixel of FIG. 15;

FIGS. 18A-18G illustrate exemplary states of the conditional-reset pixelof FIG. 15 during the operational phases shown in FIGS. 16 and 17;

FIG. 19 illustrates an alternative embodiment of a conditional-resetpixel capable of executing the conditional-reset/conditional-restorationoperations described in reference to FIGS. 16-18G;

FIG. 20 illustrates an embodiment of a conditional-reset 3-transistorpixel and read-out circuitry that permits sampling noise reductionthrough both digitally correlated double sampling and analoguncorrelated double sampling;

FIG. 21 is a flow diagram illustrating a combination of a digitallycorrelated double sampling operation with one or more analoguncorrelated double sampling operations that may be carried out toachieve a reduced-noise pixel read-out within the conditionally-resetthree-transistor pixel and read-out architecture of FIG. 20;

FIG. 22 illustrates a more detailed embodiment of the pixel architectureand read-out circuit of FIG. 20;

FIG. 23 illustrates an alternative embodiment of the pixel architecturepresented in FIGS. 20 and 22;

FIG. 24 illustrates exemplary residue-mode and inter-frame integrationread-outs with respect to varying light intensities;

FIG. 25 illustrates an exemplary per-pixel frame processing approachthat may be employed within a still or video imaging system, leveragingthe inter-frame integration approach shown in FIG. 24 to yieldrelatively high-SNR images in low light conditions;

FIG. 26 illustrates an embodiment of an imaging system capable ofgenerating image frames using the inter-frame integration approachoutlined in FIGS. 24 and 25;

FIG. 27 illustrates an exemplary subframe organization and timecodeassignment that may employed within the imaging system of FIG. 26;

FIG. 28 illustrates an exemplary approach to estimating the output framevalues for pixels which yield no non-zero sample values (i.e., no resetevents) and thus are “coasting” during a given frame;

FIG. 29 illustrates an exemplary frame processing sequence that may beexecuted by the ISP of FIG. 26 to implement the inter-frame integrationtechniques described in reference to FIGS. 24-28;

FIG. 30 contrasts the dynamic ranges and SNR achieved in imagingsimulations with and without the inter-frame integration described inreference to FIGS. 24-29; and

FIG. 31 illustrates an exemplary subframe organization for a still-framecapture mode that includes an extended exposure time.

DETAILED DESCRIPTION

In some image sensors, electrical information representing a photonresponse and resulting from light incident upon a pixel region (referredto herein as a “pixel signal”) is converted to a digital image datavalue by read circuitry. The read circuitry can reside within the imagesensor, or can be located external to the image sensor. In someapproaches, a read circuit can be located within the image sensor foruse by one or more pixel regions adjacent or near the read circuit. Forread circuits located external to the image sensor, the pixel signals ofone or more pixel regions associated with the read circuits can betransferred from the pixel regions to the read circuits.

Each read circuit samples a pixel region, receives a pixel signal fromthe sampled pixel region, and converts the pixel signal to a multi-bitdigital value representative of the pixel signal. In the event that apixel signal or a digital value representative of the pixel signalexceeds a sampling threshold, the pixel signal stored at the pixelregion associated with the pixel signal is reset (for instance, byresetting a photosensitive element associated with the pixel region). Ifthe pixel signal or the digital value do not exceed the samplingthreshold, the pixel signal stored at the pixel region is not reset. Thesampling of a pixel region and the resetting of a pixel signal at thepixel region only when the pixel signal exceeds a sampling threshold isreferred to herein as “non-destructive sampling with conditional reset.”

Image Sensor Overview

FIG. 1 illustrates a partial cross-section of an image sensor 25 usefulin an embodiment. In image sensor 25, light passing through a microlensarray 10 and a color filter array 12 (useful for color imaging) isincident upon a silicon section 20 of the image sensor. The use ofmicrolenses (or other concentrating optics) and color filters isoptional and is shown here for illustrative purposes only. Silicon 20contains photodiodes (not shown) to collect charge generated by photonsabsorbed by the silicon, and access transistors (also not shown) tooperate the photodiodes. Pixel array IC wiring 14 provides connectionsused to route signals and supply voltages within the array. As shown,image sensor 25 is a BackSide Illuminated (BSI) sensor because lightenters the silicon from the side of the integrated circuit opposite thewiring layers and primary active circuit formation. Optionally, pixelarray IC wiring 14 can be arranged between the color filter array 12 andsilicon 20 (with primary active circuit formation within the “top” ofthe silicon as oriented in FIG. 1) for FrontSide Illumination (FSI).

The image sensor 25 includes a plurality of IPs (“image pixels”),IP1-IP3 shown, upon which light collected by the lenses of the microlensarray 10 is respectively incident. Each IP includes one or morephotodiodes embedded within the silicon 20. At least some photonsentering silicon 20 are converted to electron-hole pairs in the siliconand the resulting electrons (or holes in alternate embodiments) arecollected by the IPs. The description herein will refer to this processas the capture and conversion of light by the IPs into image data forthe purposes of simplicity. Each IP of the image sensor represents aportion of the surface area of the image sensor, and the IPs of theimage sensor may be organized into various arrays of columns and rows.In a CMOS or CCD image pixel technology, each IP (for instance, eachphotosensor) converts light incident upon the IP into a charge andincludes readout circuitry configured to convert the charge into avoltage or current. In one embodiment, the light captured by each IP ofthe image sensor represents one pixel of image data for an associateddigital image, though in other embodiments image data from multiple IPsis combined to represent a fewer number (one or more) of pixels(downscaling).

The image sensor 25 may include components outside the IP array.Similarly, portions of the IP array may include components that do notconvert light into charge. The region defined by the IPs in theaggregate will be referred to as the image sensor region. As describedherein, the image sensor may include amplifiers, analog-to-digitalconverters (“ADCs”), comparators, controllers, counters, accumulators,registers, transistors, photodiodes, and the like. In differentarchitectures, some of these components may be located within the imagesensor region or external to the image sensor region, and somecomponents may be located on a companion integrated circuit. In theseembodiments, a lens (such as those of the microlens array 10) may beconfigured to direct light toward the actual light-sensing elementswithin the IP rather than, for example, on the amplifiers, comparators,controllers, and other components.

As noted above, an image sensor may include an array of multiple IPs.Each IP, in response to light (for instance, one or more photons),captures and stores a corresponding charge. In one embodiment, uponsampling an IP, if a pixel signal representative of the charge stored atthe IP exceeds a sampling threshold, the pixel signal is converted to adigital value representing the pixel signal and the charge stored by theIP is reset. Alternatively, upon sampling an IP, a pixel signalrepresentative of the charge stored at the IP is converted to a digitalvalue representative of the pixel signal, and if the digital valueexceeds a sampling threshold, the charge stored by the IP is reset. Inother embodiments, an analog-to-digital conversion is begun, and whenenough of the conversion has been completed to determine whether thethreshold is exceeded, a determination is made as to whether to continuethe conversion. For instance, in a successive approximation register(“SAR”) ADC, if the threshold is equal to a most-significant-bit(s)pattern, as soon as the pattern is resolved a determination can be madeas to whether to continue the conversion and perform a reset of thepixel, or stop the conversion. A determination of whether a pixel signalor a digital value representative of a pixel signal exceeds a samplingthreshold can be made through the use of a comparator configured tocompare the pixel signal or the digital value to a sampling threshold.

FIG. 2 illustrates an analog pixel image sensor with multiple pixelsignal thresholds, according to one embodiment. The image sensor of FIG.2 is a CMOS sensor, and includes an IP array 40. The IP array caninclude any number of columns and rows, with any number of IPs percolumn and per row. IP column 50, a column representative of full orpartial IP columns in the IP array, is highlighted in FIG. 2. The IPcolumn 50 includes a plurality of IPs communicatively coupled via thecolumn line 55. IP 60, an IP representative of IPs in the IP array, ishighlighted in FIG. 2.

The IP 60 includes a photo diode 65 together with control elements thatenable the photo diode to be precharged in preparation for exposure andthen sampled after exposure. In operation, a transistor 70 is switchedon to couple the cathode of the photo diode to a voltage source and thus“precharge” the cathode of the photo diode to a precharge voltage. Thetransistor 70 is switched off at or before the start of an exposureinterval. With the transistor 70 off, the cathode voltage incrementallydischarges in response to photon strikes, lowering the photo diodepotential, VDET, in proportion to the amount of light detected. At theconclusion of the exposure interval, an access transistor 72 is switchedon to enable a signal representative of the photo diode potential to beamplified/driven onto the column line 55 via follower-transistor 74 aspixel signal 80.

An ADC 85 is communicatively coupled to the IP column 50 via the columnline 55. In the embodiment of FIG. 2, the ADC is located at the edge ofthe pixel array 40, and may be located within or external to the imagesensor on which the IP array is located. The ADC receives the pixelsignal 80 (the representation of the analog photo diode potential) fromthe IP 60. The ADC digitizes the pixel signal to generate a 3-bitdigital value (“Pix[2:0]”) representative of the pixel signal. The ADCincludes 7 pixel thresholds, Threshold 1 to Threshold 7 (referred toherein as “VT1 to VT7”). If the magnitude of the pixel signal is lessthan Vpre but greater than VT1, the ADC converts the pixel signal to thedigital value “000”. Pixel signals less than VT1 but greater than VT2are converted to the digital value “001”, pixel signals between VT2 andVT3 are converted to “010”, and so forth, up to pixel signals less thanVT7, which are converted to “111”.

In the embodiment of FIG. 2, the potential difference between successivepixel thresholds is approximately the same (e.g., VT3−VT4≈VT5−VT6). Inother words, the pixel thresholds are linearly distributed between VT1and VT7. In addition, in the embodiment of FIG. 2, the potentialdifference between Vpre and VT1 is greater than the potential differencebetween successive pixel thresholds (e.g., Vpre−VT1>VT3−VT4), althoughin other embodiments all steps are equal. The selection of VT1 such thatVpre−VT1>VT3−VT4 reduces the effect of, e.g., dark noise when samplingan IP. The potential difference between VT7 and Vfloor in the embodimentof FIG. 2 also can be greater than the potential difference betweensuccessive pixel thresholds (e.g., VT7−Vfloor>VT3−VT4). Finally, insteadof linear threshold spacing, a given embodiment can space the thresholdsexponentially, e.g., with each threshold spacing doubling from the onebelow. For systems that accumulate multiple ADC samples to form animage, exponential spacing is converted to a linear value prior toaccumulation.

Vfloor represents the pixel saturation threshold at which the cathodevoltage of the photo diode 65 no longer linearly discharges in responseto photon strikes. For pixel signals within the linear sensitivityregion 90, the conversion of pixel signals to digital values is shown ingraph 95. It should be noted that the maximum number of detectablephoton strikes (i.e., the pixel saturation point) is proportional to thecapacitance of the photo diode and thus its physical size. Consequently,in a traditional sensor design the photo diode footprint is dictated bythe dynamic range required in a given application and does not scaleappreciably with shrinking process geometries.

During the capture of an image, in one embodiment the IPs of a given rowor rows in the IP column 50 and each other column in the IP array 40 aresuccessively sampled and the pixel signals associated with each areconverted into digital values using the ADC or ADCs associated with eachcolumn. The digital values output by the ADCs are accumulated(conditionally in some embodiments, as explained below) and storedduring the image capture period. Other types and configurations of IPsthan that illustrated in FIG. 2 can be used in the image sensor system.For instance, a different arrangement of transistors can be used thanthe transistors 70, 72, and 74. In addition, although one ADC 85 isshown in FIG. 2 in conjunction with the IP column 50, in otherembodiments, more than one ADC can be used per IP column, with differentADC groups serving different sections of the array rows of the ADCcolumn. Additional combinations of ADCs (in the form of read circuits)and IPs are described below in greater detail. Finally, the output ofthe ADC (e.g. Pix[2:0] in the embodiment of FIG. 2) can be any multi-bitlength, and can be associated with any number of thresholds distributedin any manner between V_(pre) and V_(floor).

Image Sensor System with Multi-Bit Sampling and Conditional Reset

FIG. 3 illustrates an example image sensor read circuit configured toconvert a pixel signal into a multi-bit digital conversion, according toone embodiment. The embodiment of FIG. 3 illustrates an IP 100, an IPmemory 116, and a read circuit 110, the read circuit including anADC/comparator circuit 112 (hereinafter “ADC/comparator”) and an adder114. It should be noted that in other embodiments, the modules of FIG. 3can include additional, fewer, and/or different components. For example,the ADC/comparator can be implemented as separate components, and theadder can be located external to the read circuit.

The IP 100 represents an IP in an image sensor, and can be, forinstance, the IP 60 of FIG. 2. The IP 100 receives one or more controlsignals, for instance from external control logic. A control signal canenable the IP to begin an image capture, for instance by resetting theIP to Vpre and enabling the exposure of the IP's photosensitive elementto light to result in the storing of charge relative to Vpre. Similarly,a control signal can enable the IP to end an image capture, for instanceby disabling the exposure of the IP's photosensitive element to lightafter the passing of an image capture period. A control signal can alsoenable the outputting of a pixel signal by an IP and the subsequentconversion of the pixel signal to a digital value representative of thepixel signal by a read circuit (referred to herein as “sampling the IP”or “sampling the pixel signals”). As described above, a pixel signal canbe a representation of the integrated charge (e.g., a source followervoltage, an amplified voltage, or a current having a componentproportional to the integrated charge).

The IP 100 receives a reset signal, for instance from external controllogic. The reset signal resets the charge stored by the IP to Vpre, forinstance at the beginning of an image capture period. The IP alsoreceives a conditional reset signal from the ADC/comparator 112 (in somecircuits, the conditional reset and initial reset are supplied usingcommon circuitry). The conditional reset signal resets the charge storedby the IP, for instance during an image capture period in response to apixel signal exceeding a sampling threshold when the IP is sampled. Itshould be noted that in other embodiments, the conditional reset signalis received from a different entity. In one implementation, theADC/comparator may determine that the pixel signal exceeds a samplingthreshold, and may enable external control logic to output a conditionalreset signal to the IP; in such an embodiment, the reset signal (arow-wise signal) and the conditional reset signal (a column-wise signal)may be ANDed by the IP to initiate all resets. For simplicity, theremainder of the description will be limited to embodiments in which theADC/comparator provides conditional reset signals to the IP.

The read circuit 110 receives a threshold signal, a sample signal (or“sample enable signal”), a compare signal (or “compare enable signal”),a residue signal (or “residue enable signal”), and a reset signal, forinstance from external control logic, and receives pixel signals fromthe IP 100. The IP memory element 116 corresponding to IP 100 receives areadout signal that selects it for readout/write by adder 114 and forexternal readout. The ADC/comparator 112 samples the IP 100 in responseto receiving one or more sample signals. During an image capture, theADC/comparator receives a sample signal at various sampling intervals,for instance periodically or according to a pre-defined samplinginterval pattern (referred to herein as a “sampling policy”).Alternatively, the sample signal received by the ADC/comparator caninclude a sampling policy, and the ADC/comparator can be configured tosample the IP based on the sampling policy. In other embodiments, the IPreceives one or more sample signals and outputs pixel signals based onthe received sample signals. In yet other embodiments, the IP outputspixel signals periodically or according to a sampling policy, or theADC/comparator samples pixel signals periodically or according to asampling policy, independent of received sample signals. TheADC/comparator can request a pixel signal from the IP prior to samplingthe pixel signal from the IP.

During a sampling of the IP, the ADC/comparator 112 receives a pixelsignal from the IP and converts (optionally in some embodiments based onthe pixel signal exceeding the sampling threshold) the pixel signal to amultiple-bit digital value representative of the pixel signal. If thepixel signal exceeds a sampling threshold, the ADC/comparator outputs aconditional reset signal to reset the charge stored at the IP. If thepixel signal does not exceed a sampling threshold, the ADC/comparatordoes not output a conditional reset signal to reset the charge stored atthe IP. The sampling threshold can be varied during the image captureand received via the threshold signal, or can be pre-determined orpreset for a given image capture. One sampling threshold can be usedduring multiple image captures, different sampling thresholds can beused for different image captures, and multiple sampling thresholds canbe used during a single image capture. In one embodiment, the samplingthreshold varies in response to detected changing light conditions (forinstance, the sampling threshold can decrease in response to low lightconditions, and can increase in response to high light conditions).

In one embodiment, the sampling threshold is an analog signal threshold.In this embodiment, the ADC/comparator 112 includes an analog comparatorand compares the pixel signal to the sampling threshold to determine ifthe pixel signal exceeds the sampling threshold. If the pixel signalincludes a voltage representative of the charge stored by the IP 100,the sampling threshold is exceeded if the pixel signal is lower than thesampling threshold. Using the embodiment of FIG. 2 as an example, if thesampling threshold of the ADC/comparator is Threshold 4, then the pixelsignal will exceed the sampling threshold only if the pixel signalincludes a voltage lower than the voltage associated with Threshold 4.

In one embodiment, the sampling threshold is a digital signal threshold.In this embodiment, the ADC/comparator 112 includes a digitalcomparator, and first converts the pixel signal to a digital valuerepresentative of the pixel signal. The ADC/comparator then compares thedigital value to the sampling threshold to determine if the pixel signalexceeds the sampling threshold. Using the embodiment of FIG. 2 as anexample, for a sampling threshold of “101”, if the ADC/comparatorconverts a pixel signal to a digital value of “001” (indicating that thepixel signal is between Threshold 1 and Threshold 2), then the pixelsignal does not exceed the sampling threshold and a conditional resetsignal is not outputted. However, if the ADC/comparator converts a pixelsignal to a digital value of “110” (indicating that the pixel signal isbetween Threshold 6 and Threshold 7), then the pixel signal does exceedthe sampling threshold and a conditional reset signal is outputted.

In another embodiment, the sampling threshold is a digital signalthreshold that can be evaluated prior to the complete digital conversionof the pixel signal. This can be advantageous in some embodiments or usecases to allow faster conditional reset of a pixel, and/or power savingsby avoiding unneeded ADC operations. For instance, with a successiveapproximation register ADC, multiple clock cycles are used to resolvethe digital representation of the pixel signal. The first clock cycleresolves the most significant bit, the second clock cycle the next mostsignificant bit, etc., until all bit positions have been resolved. Usingthe embodiment of FIG. 2 as an example, for a sampling threshold of“100,” a determination of whether the threshold is met or not can beresolved after the first SAR ADC clock cycle. For a sampling thresholdof “110,” a determination of whether the threshold is met or not can beresolved after the second SAR ADC clock cycle. For embodiments with abit depth of, e.g., 6 or 8 bits, making a reset determination after oneor two conversion cycles can result in significant time/power savings,which can be realized by selecting a sampling threshold with one or moreLSBs that are 0.

In one embodiment, a row-wise compare signal is supplied to eachADC/comparator “compare” signal input, and signals the ADC/comparator asto the appropriate clock cycle to perform the comparison. When thecompare signal is asserted, the comparison is performed based on thecurrent state of the analog-to-digital conversion. If the threshold ismet by the compare for ADC/comparator 112, the conditional reset signalis asserted to IP 100 and to adder 114, and the SAR ADC continuesconverting the pixel signal. If the threshold is not met, theconditional reset signal is not asserted, and can be used in conjunctionwith the compare signal to gate the clock signal of SAR ADC to terminatethe conversion.

The ADC/comparator 112 outputs a digital value representative of a pixelsignal received by the ADC/comparator (referred to herein as a “digitalconversion”) to the adder 114. The ADC/comparator 112 can output adigital conversion in response to the pixel signal associated with thedigital conversion exceeding a sampling threshold. The conditional resetsignal can be used as an enable to signal to the adder 114 to load thedigital conversion and add it to the IP memory 116 locationcorresponding to IP 100 (which in this embodiment is selected from aplurality of such locations by address selection of the readout line).In other embodiments, the ADC/comparator outputs a digital conversionduring each sampling of the IP 100, regardless of whether the pixelsignal associated with the digital conversion exceeds a samplingthreshold. In these embodiments, the adder can be configured toaccumulate digital conversions associated with pixel signals that exceeda sampling threshold and to disregard digital conversions associatedwith pixel signals that do not exceed a sampling threshold. Alternately,if the threshold is set to “001” in FIG. 2, for example, the adder canunconditionally add the digital conversion to IP memory 116 each time IP100 is read, while still producing correct results.

In one embodiment, the ADC/comparator 112 also outputs a digitalconversion in response to receiving a residue signal assertion (withoutthe compare signal being asserted). The residue signal assertion isassociated with the end of an image capture, and enables theADC/comparator to output a full digital conversion to the adder 114regardless of whether the pixel signal associated with the digitalconversion exceeds a sampling threshold, and asserts the conditionalreset. The residue signal can prevent the loss of image informationassociated with light received by the IP 100 but not surpassing thethreshold at the end of a capture period. If the pixel signalrepresentative of such received light does not exceed the samplingthreshold, the ADC/comparator otherwise may not output the digitalconversion associated with the pixel signal, and the charge stored bythe IP would not be reset by the conditional reset signal (which is alsotriggered by assertion of the residue signal). In embodiments where theADC/comparator outputs digital conversions to the adder regardless ofwhether the pixel signals associated with the digital conversions exceeda sampling threshold, the adder can receive the residue signal, and canbe configured to accumulate a digital conversion associated with a pixelsignal received at the end of a capture period in response to receivingthe signal.

The adder 114 is configured to accumulate digital conversions receivedduring a capture period. As discussed above, in embodiments in which theADC/comparator 112 outputs digital conversions only if the pixel signalsassociated with the digital conversions exceed a sampling threshold, theadder accumulates all received digital conversions (including theadditional digital conversion output by the ADC/comparator in responseto receiving a residue signal) into IP memory 116. In embodiments inwhich the ADC/comparator outputs digital conversions associated witheach received pixel signal, the adder accumulates only the digitalconversions associated with pixel signals that exceed the samplingthreshold, plus the digital conversion output by the ADC/comparator inresponse to receive a residue signal, into IP memory 116; suchembodiments require the adder to be aware of when pixel signals exceed asampling threshold and when a residue signal is received, and are notdiscussed further herein for the purpose of simplicity.

The adder 114 receives reset/add control signaling, for instance fromexternal control logic. In response to receiving a reset signal (forinstance at the beginning of an image capture period), the accumulatorstores all zeros to the selected IP memory location 116 the accumulationof received digital conversions as image data. The adder also receives areset signal and resets the accumulation of received digitalconversions.

In alternative embodiments, the adder is located external to the readcircuit 110. For instance, the ADC/comparator can output a stream ofconversions to a digital channel (e.g., multiplexed with otherconversions from other ADCs) to a separate circuit that supplies theaccumulation function. In such a case, the ADC/comparator must alsooutput a symbol for “no conversion,” which can be 0. One possibility isfor a circuit in the digital channel interface (e.g., PHY 134 in FIG. 4)to code digital conversions to reduce bandwidth. A “no conversion” inone embodiment is output as a “00,” a upper threshold exceeded ADCconversion is output as a “01,” and all other ADC conversions are outputas “1xxxxxx,” where an x represents one of the resolved bits of the ADCconversion and the number of x positions is equal to the bit depth ofthe ADC.

In one embodiment, the IP is configured to output a pixel signal andreceive a conditional reset on the same line. In this embodiment, the IPand the ADC/comparator 112 alternately drive the pixel signal and theconditional reset on the shared line. For example, the IP can output apixel signal on the shared line during a first portion of a sampleperiod, and can receive conditional resets on the shared line during asecond portion of a sample period. Finally, the ADC/comparator canreceive a threshold signal, a sample signal, and a residue signal on ashared line. For example, the ADC/comparator can receive a thresholdsignal at the beginning of an image capture, can receive sample signalsthroughout the image capture period, and can receive a residue signal atthe end of the image capture period. It should also be noted that thereset signal received by the IP can be the same reset signal received bythe accumulator 114, and can be received on a shared line.

FIG. 4 illustrates an example embodiment of an image sensor system witha multi-bit architecture, according to one embodiment. The image sensorsystem 120 of FIG. 4 includes an image sensor region 125, a read circuitarray 130, control logic 132, and a physical signaling interface 134. Inother embodiments, the image sensor system may include fewer,additional, or different components than illustrated in the embodimentof FIG. 4 (for instance, the circuit may have memory 116 integratedtherewith). The image sensor system shown in FIG. 4 can be implementedas a single IC, or can be implemented as multiple ICs (for instance, theimage sensor region and the read circuit array can be located onseparate ICs). Further, various components (such as the read circuitarray, the control logic, and the physical signaling interface) can beintegrated within the image sensor region 125.

For purposes of example, the image sensor system 120 and a host IC (notshown in FIG. 4) communicatively coupled to the image sensor system areassumed to form the primary image acquisition components within a camera(e.g., a still-image or video camera within a mobile device, compactcamera, digital SLR camera, stand-alone or platform-integrated webcam,high-definition video camera, security camera, automotive camera, etc.).The image sensor IC and host IC can be more generally deployed alone ortogether with like or different imaging components within virtually anyimaging system or device including without limitation metrologyinstruments, medical instruments, gaming systems or other consumerelectronics devices, military and industrial imaging systems,transportation-related systems, space-based imaging systems and soforth. Operation of the image sensor system generally involves thecapture of an image or frame through the exposure of IPs to light, theconversion of stored charge as a result of the exposure into image data,and the outputting of the image data to a storage medium.

The image sensor region 125 includes an IP array 127 including N-rows(indexed from 0 to N−1) and M-columns (indexed from 0 to M−1). Thephysical signaling interface 134 is configured to receive commands andconfiguration information from a host IC (e.g., a general-purpose orspecial-purpose processor, application-specific integrated circuit(ASIC) or any other control component configured to control the imagesensor IC), and is configured to provide the received commands andconfiguration information to the control logic 132. The physicalsignaling interface is also configured to receive image data from theread circuit array 130 and to output received image data to the host IC.

The control logic 132 is configured to receive commands andconfiguration information from the physical signaling interface 134, andis configured to transmit signals configured to manipulate theoperations and functionality of the image sensor system 120. Forexample, in response to receiving a command to capture an image orframe, the control logic may output a series of exposure signals(configured to cause IPs to reset) and sample signals (configured tocause the read circuits in the read circuit array 130 to sample thepixel signals from the IPs in the IP array 127), enabling the capture ofthe image or frame by the image sensor system. Similarly, in response toreceiving a command to initialize or reset the image sensor system, thecontrol logic may output reset signals configured to reset each IP inthe IP array, causing each IP to disregard any accumulated charge. Thecontrol signals produced by the control logic identify particular IPswithin the IP array for sampling, may control the functionality of readcircuits associated with IPs, or may control any other functionalityassociated with the image sensor system. The control logic is shown inFIG. 4 as external to the image sensor region 125, but as noted above,all or portions of the control logic may be implemented locally withinthe image sensor region.

The control logic 132 outputs control and reset signals for each IP inthe image sensor region 125. As illustrated in the embodiment of FIG. 4,each IP in an image pixel IP[X] [Y] receives a row-parallel Cntrl[X]signal (corresponding to a “row” select control signal for each IP) anda row-parallel Reset[X] signal from the control logic to reset the IPs,wherein “X” and “Y” refer to the coordinates of the IP within the imagesensor region. Although the control signal and reset signals received atany given IP are each only 1 bit as indexed in the embodiment of FIG. 4,it is to be appreciated that such an indexing is done for the purposesof simplicity only, and that these signals may in practice be any widthor dimension.

The read circuit array 130 includes M read circuits, each configured toreceive pixel signals from a column of IPs in the IP array 127. Itshould be noted that in other embodiments, the read circuit array caninclude multiple read circuits configured to receive pixel signals fromeach IP column, as is discussed in FIGS. 5a, 5b, and 5c . A pixel signalbus couples the IPs in each IP column in the IP array to the readcircuit associated with the IP column within the read circuit array.Each IP is configured to output a pixel signal produced by the IP to thepixel signal bus, and each read circuit is configured to sample thepixel signals from the IPs in the IP column associated with the readcircuit. For example, read circuit 0 is configured to sample pixelsignals from pixel signal bus 0, and so forth. Each read circuit in theread circuit array can sample pixel signals iteratively from IPs in theIP column associated with the read circuit (for instance, by samplingpixel signals from successive IPs in order over multiple passes), or cansample pixel signals according to a pre-determined non-sequential order.In one embodiment, read circuits can sample multiple pixel signalssimultaneously. Although not illustrated in the embodiments of FIG. 3and FIG. 4, the read circuits can additionally include memoriesconfigured to store accumulated digital values prior to outputting theaccumulated values as image data.

A conditional reset bus couples the IPs in each IP column in the IParray 127 to the read circuit associated with each IP column. Aftersampling a pixel signal from an IP in an IP column, the read circuitassociated with the IP column produces a conditional reset signal if thesampled pixel signal exceeds a sampling threshold. For example, if an IPin an IP column outputs a pixel signal to a read circuit associated withthe IP column via the pixel signal bus coupling the IP to the readcircuit, and if the read circuit determines that the pixel signalexceeds a sampling threshold, the read circuit outputs a conditionalreset signal to the IP via the conditional reset bus coupling the readcircuit to the IP and the IP resets the charge stored at the IP. Asdescribed above, the pixel signal bus and the conditional reset bus canbe implemented in a shared bus with Cntrl[X] enabling pixel signals tobe output from row X to the shared bus and Reset[X] enabling conditionalreset for pixels in row X from the shared bus, though such embodimentsare not described further herein for the purposes of simplicity.

The control logic 132 produces read control signals for the readcircuits in the read circuit array 130. The read control signals cancontrol the sampling of pixel signals from the IPs in the IP array 127by the read circuits, the conversion of sampled pixel signals intodigital values, the accumulation of the digital values, the outputtingof the accumulated digital values, and the resetting of the adders. Theread control signals can include a threshold signal, a sample signal, acompare signal, a residue signal, a readout signal, and a reset/addsignal for each read circuit in the read circuit array as described inFIG. 3.

The control logic 132 is configured to produce read control signals forthe read circuit array 130 to enable the capture of an image over animage capture period. Prior to the image capture period or at the firstuse of a particular IP memory location for an image capture period, thecontrol logic can produce a reset to cause the accumulator of each readcircuit 110 to reset the IP memory location. At the beginning of theimage capture period, the control logic can produce a threshold signalfor each of the read circuits; as discussed above, the threshold signalis used by each read circuit to determine a threshold to which pixelsignals are compared for the purposes of conditionally resetting IPsassociated with the pixel signals and accumulating digital valuesassociated with the pixel signals. During the image capture period, thecontrol logic can produce a series of sample signals configured toenable the read circuits to sample pixel signals from IPs associatedwith the read circuits. In one embodiment, the control logic producessample signals according to one or more sampling policies. Samplingpolicies are described in greater detail below. At the end of the imagecapture period, the controlled logic produces a residue signalconfigured to enable each read circuit to accumulate a digital valuerepresentative of a pixel signal regardless of whether the pixel signalexceeds a sampling threshold. After the image capture period, thecontrol logic produces a readout signal configured to enable each readcircuit to output the accumulated digital values representative ofsampled pixel signals that exceed an associated sampling threshold asimage data. The control logic may also produce a reset signal after eachimage capture period to reset the accumulated digital values within eachread circuit.

The control logic may also be configured to produce pause and resumesignals configured to cause the IPs and the read circuits to pause andresume an image capture, and to produce any other signal necessary tocontrol the functionality of the IPs and read circuits in the readcircuit array. For each read circuit, the image data output by the readcircuit is a digital representation of the light captured by each IP inan IP column associated with the read circuit. The image data isreceived by the physical signaling interface for subsequent output to ahost IC.

FIG. 5 illustrates an example image sensor system architecture with readcircuit arrays located peripherally to an IP array, according to oneembodiment. In the architecture of FIG. 5, six read circuit arrays (140a, 140 b, 140 c, 140 d, 140 e, and 140 f) are located around an imagesensor region 145 including an IP array. Unlike the embodiment of FIG.4, in which one read circuit array 130 is located to one side of theimage sensor region 125, the read circuit arrays 140 of FIG. 5 arelocated on all sides of the image sensor region 145. The read circuitarrays can be located within an IC also containing the image sensorregion, or can be located on one or more separate ICs. For example, eachread circuit array could be located on the periphery of an image sensorIC, or could be located in dedicated read circuit array ICs locatedadjacent to the image sensor IC.

In the previous embodiment of FIG. 4, each read circuit in the readcircuit array 130 is coupled to an IP column in the IP array 127. In theembodiment of FIG. 5, each read circuit array 140 x is coupled to a setof six IPs from partial rows and partial columns of the image sensorregion 145. For example, read circuit array 140 a is coupled to IP1,IP2, IP3, IP7, IP8, and IP9. Each read circuit array 140 x includes oneor more read circuits. In one embodiment, each read circuit arrayincludes 6 read circuits, with each read circuit in a read circuit arraycoupled to one IP. In such an embodiment, each read circuit samples onlythe IP to which it is coupled. More typically, each read circuit will beshared by a block of IPs comprising a large number of rows and one ormore columns. Although control logic is not illustrated in theembodiment of FIG. 5, each read circuit array can be coupled touniversal control logic, or each may be coupled to dedicated controllogic. Further, although a physical signaling interface is notillustrated in the embodiment of FIG. 5, each read circuit array mayoutput image data via a common bus to a common physical signalinginterface, or may output image data via a dedicated bus to a dedicatedphysical signaling interface coupled to each read circuit array.

FIG. 6a illustrates a top view of a pixel array IC in an exampletwo-layer image sensor system architecture, according to one embodiment.The pixel array IC of FIG. 6a includes peripheral circuitry 162surrounding an IP array. The IP array includes row control circuitry 164and four row groups of IPs (IP Row Groups 0 through 3). Each IP rowgroup is the width of the array and includes one-fourth of the rows inthe array, and the row control circuitry provides control and resetsignals needed for operation of the IPs (for instance, signalsconfigured to cause the IPs to be enabled for reset and selected forreadout, and any other signals discussed herein).

FIG. 6b illustrates a top view of a preprocessor IC in an exampletwo-layer image sensor system architecture, according to one embodiment.The preprocessor IC of FIG. 6b includes peripheral circuitry 172surrounding a read circuit array. The read circuit array includes aphysical signaling interface 175 (which may alternately be on pixelarray IC 160), read control circuitry 176, four read circuit arrays(read circuit array 0 through 3), and accompanying memory groups 0A/B,1A/B, 2A/B, and 3A/B. Each read circuit array includes one or more readcircuits (including an ADC, adder, and reset logic for each IP column)connected to corresponding rows in an associated memory group. When aparticular IP row is selected in an IP row group of the pixel array IC,a corresponding row in the corresponding memory group is selected on thepreprocessor IC.

FIG. 6c illustrates a cross section of the pixel array IC of FIG. 6a andthe preprocessor IC of FIG. 6b in an example two-layer image sensorsystem architecture, according to one embodiment. In the embodiment ofFIG. 6c , the pixel array IC 160 is located above the preprocessor IC170 such that the bottom surface of the pixel array IC is coupled to thetop surface of the preprocessor IC. A microlens array 180 and a colorfilter array 182 are located above the pixel array IC. The pixel arrayIC and the processor IC are coupled via pixel array IC wiring 184 andpreprocessor IC wiring 186. By locating the pixel array IC above thepreprocessor IC, the die size and percentage of surface area in theimage sensor system capable of capturing light is increased. Forinstance, in a single-layer IC architecture including an IP array andone or more read circuit arrays, the portion of the single-layer ICincluding the one or more read circuit arrays are incapable of capturinglight; such an embodiment reduces the percentage of silicon die used tocapture light incident upon the single-layer IC. This requires thecamera module footprint to be larger than the lens and the imagingarray, and increases the cost and size of the camera module. Thetop-layer of the embodiment of FIG. 6c , in contrast, does not includeread circuit arrays, so the die size of the top single layer IC isreduced to approximately the size of the IP array. Light incident uponthe top-layer passes through the microlens array and the color filterarray, is captured by the IPs in the IP array, and signalsrepresentative of the captured light are sampled by the read circuitarrays via the pixel array IC wiring and the preprocessor IC wiring.

FIG. 7 illustrates the operation of an image sensor read circuit, suchas the read circuit of FIG. 3, according to one embodiment. In theexample embodiment of FIG. 7, an image is captured over the course of 16sampling intervals. The ADC of the example embodiment of FIG. 7 convertspixel signals to 5-bit digital values, and the accumulator accumulates5-bit digital values into a 9-bit digital value during the image captureperiod. Further, in the embodiment of FIG. 7, the ADC converts receivedpixel signals into digital values representing the pixel signals suchthat each additional photon detected by an IP results in an increase inthe digital value by one. For example, if an IP detects 5 photons afterbeing reset, the pixel signal produced by the IP will be converted bythe ADC into the value “00101”. It should be emphasized that in otherembodiments, the ADC converts received pixel signals into digital valuesrepresenting the pixel signals such that multiple additional photonsdetected by an IP results in an increase in the digital value by one. Inthe embodiment of FIG. 7, pixel signals are analog voltages, and thusaren't shown in FIG. 7 for the purposes of simplicity.

At the beginning of the image capture period (sampling interval 0), acontrol signal is received configured to configure an IP of the readcircuit to be reset and begin exposure. In the embodiment of FIG. 7, the“begin exposure” control signal also resets the value stored at thememory element corresponding to the IP to zero. In addition, a thresholdsignal is received to set the sampling threshold for the read circuit ata pixel signal equivalent to 20 photons.

During the first sampling interval, 4 photons are detected by the IP.The IP then produces a pixel signal representing the charge collected bya photosensitive element within the IP equivalent in response todetecting the 4 photons, and the ADC converts this pixel signal to thedigital value “00100”. Since the 4 detected photons do not trigger thesampling interval of 20 photons (“10100”), the accumulator does notaccumulate the digital value “00100”, and the charge stored by the IP isnot dissipated (the IP is not reset). Note that the column “Photons(det.-accum.)” indicates first the number of photons detected by the IPduring a particular sampling interval and second the number ofaccumulated photons since the last conditional reset of the IP.

During sampling interval 2, 7 additional photons are detected by the IP.The charge stored by the IP increases from the charge produced inresponse to detecting 4 photons during sampling interval 1 to a chargeproduced in response to detecting 11 accumulated photons (4 photonsduring sampling interval 1 and 7 photons during sampling interval 2).The pixel signal produced by the IP in response to the stored charge isconverted to the digital value “01011”. Since the total 11 photons donot trigger the sampling threshold of 20 photons, the accumulator doesnot accumulate the digital value “01011”, and the IP is not reset.Similarly, during sampling interval 3, 2 additional photons are detectedby the IP, and the charge stored by the IP increases to a chargeproduced in response to detecting 13 accumulated photons (4 photonsduring sampling interval 1, 7 during sampling interval 2, and 2 duringsampling interval 3). The pixel signal produced by the IP in response tothis increased stored charge is converted to the digital value “01101”.Since the accumulated 13 photons do not trigger the sampling thresholdof 20 photons, the accumulator does not accumulate the digital value“01101”, and the IP is not reset.

During sampling interval 4, 11 additional photons are detected by theIP. The charge stored by the IP increases to a charge equivalent todetecting 24 accumulated photons (4 during sampling interval 1, 7 duringsampling interval 2, 2 during sampling interval 3, and 11 duringsampling interval 4). The pixel signal produced by the IP in response tothe stored charge is converted to the digital value “11000”. Since theaccumulated 24 photons exceeds the sampling threshold of 20 photons, theadder accumulates the digital value “11000” into the memory element forthe IP, and the IP is reset.

The 14 photons detected during sampling interval 5 do not exceed thesampling interval of 20, the digital value produced by the ADC, “01110”is not accumulated and the IP is not reset. The 8 photons detectedduring sampling interval 6 results in an accumulated detection of 22photons by the IP (14 photons during sampling interval 5, and 8 duringsampling interval 6), and the adder accumulates the digital value“10110” (resulting in a total accumulated value of “000101110” into thememory element), and the IP is reset.

This process is repeated for each of the 16 sampling intervals. Thedigital values produced by the ADC during sampling intervals 10, 14, and15 are all accumulated in response to the sampling threshold of 20photons being exceeded by the number of accumulated photons detected bythe IP. Accordingly, the IPs are reset for the sampling intervalsfollowing these intervals (sampling interval 11, 15, and 16). Duringsampling interval 16, 19 photons are detected by the IP, which does notexceed the sampling threshold of 20 photons. In addition, duringsampling interval 16, a residue signal is received configured toinstruct the accumulator to accumulate the digital value produced by theADC (the residue value 190, “10011”). Accordingly, the adder accumulatesthe value “10011” to the maintained accumulation value “001111011” inthe memory element to produce the image data 195, “010001110”. Finally,a reset signal is received during sampling interval 16, which enablesthe read circuit to output the image data and which resets the valuesoutput by the ADC and stored at the accumulator to zero subsequent tooutputting the image data.

FIG. 8 illustrates pixel information flow in an image capture system,according to one embodiment. During the course of an image captureperiod, an IP 200 detects photons and outputs pixel signals 202 to theread circuit. In response, the read circuit 204 converts the receivedpixel signals to digital values representative of the receive pixelsignals, and for each digital value associated with a pixel signal thatexceeds a sampling threshold, accumulates the digital value and resetsthe IP. After the image capture period, the accumulated digital valuesare output as image data 206.

A post processing module 208 receives the image data 206 and performsone or more processing operations on the image data to produce theprocessed data 210. In one embodiment, a response function can be usedto transform the image data 206 according to a desired response. Forexample, the image data can be transformed with a linear function or alogarithmic function based on the intensity of the light detected by theIP. The processed data is then stored in memory 212 for subsequentretrieval and processing. The IP 200, the read circuit 204, the postprocessing module, and the memory can be located within an IC, or can belocated within separate coupled ICs.

FIG. 9 illustrates various temporal sampling policies for use by animage sensor read circuit, such as the read circuit of FIG. 3, accordingto one embodiment. In the embodiment of FIG. 9, an image is capturedover an image capture period 220 equivalent to 16 time units. For eachof the three illustrated sampling policies, an “x” indicates thesampling of a given IP by a read circuit.

In sampling policy 1, the read circuit samples the IP after each of the16 time units. In sampling policy 2, the read circuit samples the IPafter every 4 time units. As the read circuit in sampling policy 2samples the IP less frequently than the read circuit in sampling policy1, the IP in sampling policy 2 is more likely to saturate than the IP insampling policy 1. However, the resources (processing, bandwidth, andpower) required to implement sampling policy 2 (4 total samples) may belower than the resources required to implement sampling policy 1 (16total samples), since the read circuit in sampling policy 2 samples theIP only 25% as often as the read circuit in sampling policy 1.

In sampling policy 3, the read circuit samples the IP after time units1, 2, 4, 8, and 16. The exponential spacing of the samplings of samplingpolicy 3 provide short sample intervals (for instance, the sampleinterval between time unit 0 and time unit 1) and long sample intervals(for instance, the sample interval between time unit 8 and time unit16). Allowing for both short and long sampling intervals preserves thedynamic range of sampling policy 1 with nearly as few samplings assampling policy 2 (5 samplings for sampling policy 3 vs. 4 samplings forsampling policy 2). Other sampling policies not illustrated in FIG. 9may also be implemented by read circuits in the image sensor systemsdescribed herein. Depending on the overall length of an exposureinterval or other scene- or user-dependent factors, different samplingpolicies can be selected to meet desired power, SNR, dynamic range, orother performance parameters.

High-SNR Image Sensor with Non-Destructive Threshold Monitoring

While the three-transistor (3T) pixel architecture shown in FIG. 2 issuitable for many applications, four-transistor (4T) designs having a“transfer gate” disposed between the photodiode and source follower(i.e., between node “V_(DET)” of photosensitive element 65 and element74 in FIG. 2) provide a number of advantages. First, the now-isolatedfloating diffusion at the gate of the source follower may be reset(e.g., coupled to V_(DD)) without disturbing the charge state of thephotodiode, thereby enabling a correlated double-sampling (CDS)operation in which the noise floor of the floating diffusion is sampledprior to charge integration and then subtracted from the subsequentsampling of the photodiode potential, canceling the noise andsignificantly improving the SNR. Another advantage is,counterintuitively, a more compact pixel design as the switchedconnection between the photodiode and source follower (i.e., via thetransfer gate) enables the source follower, reset and access transistorsto be shared among multiple photodiodes. For example, only seventransistors are required to implement a set of four “4T” pixels having ashared source follower, reset transistor and access transistor (i.e.,four transfer-gates plus the three shared transistors), thus effectingan average of 1.75 transistors per pixel (1.75T).

In terms of pixel read-out, the direct connection between photodiode andsource follower in a 3T pixel permits the charge state of the photodiodeto be read-out without disturbing ongoing photocharge integration. This“non-destructive read” capability is particularly advantageous in thecontext of the conditional reset operation described above as the 3Tpixel may be sampled following an integration interval and thenconditionally permitted to continue integrating charge (i.e., not bereset) if the sampling operation indicates that the charge level remainsbelow a predetermined threshold. By contrast, the charge transferbetween photodiode and floating diffusion as part of a 4T pixel readoutdisrupts the state of the photodiode, presenting a challenge forconditional-reset operation.

In a number of embodiments described below in connection with FIGS.10-14, a modified 4T pixel architecture is operated in a manner thatdissociates the reset threshold from pixel sample generation to enable anon-destructive (and yet CDS) over-threshold determination. That is,instead of reading out the net level of charge accumulated within thephotodiode (i.e., a pixel sampling operation) and conditionallyresetting the photodiode based on that read-out (i.e., as in a 3T pixelsampling operation), a preliminary over-threshold sampling operation isexecuted to enable detection of an over-threshold state within thephotodiode, with the full photodiode read-out (i.e., pixel samplegeneration) being conditionally executed according to the preliminaryover-threshold detection result. In effect, instead of conditionallyresetting the photodiode according to the pixel value obtained from fullphotodiode readout, full photodiode readout is conditioned on the resultof a preliminary, and largely non-destructive, determination of whetherthe threshold has been exceeded; an approach enabled, in at least oneembodiment, by dissociating the conditional-reset threshold from thepixel value generation.

FIG. 10 illustrates an embodiment of a modified 4T pixel 250, referredto herein as a “progressive read-out pixel,” in which a non-destructiveover-threshold detection operation is executed to enableconditional-reset operation in conjunction with correlated doublesampling. As explained more fully below, the over-threshold detectioninvolves a limited read-out of the photodiode state which, whendetermined to indicate an over-threshold condition, will trigger a morecomplete read-out of the photodiode state. That is, pixel 250 isread-out in a progression from a limited over-threshold detectionread-out to a complete read-out (the latter being conditional accordingto the over-threshold detection result) and is thus referred to hereinas a progressive-readout pixel.

Still referring to FIG. 10, progressive read-out pixel 250 includes atransfer gate 251 disposed between a photodiode 260 (or any otherpracticable photosensitive element) and floating diffusion node 262, anda transfer-enable transistor 253 coupled between a transfer-gate rowline (TGr) and the control terminal (e.g., gate) of transfer gate 251.The gate of transfer-enable transistor 253 is coupled to a transfer-gatecolumn line (TGc) so that, when TGc is activated, the potential on TGris applied (minus any transistor threshold) via transfer-enabletransistor 253 to the gate of transfer-gate 251, thus enabling chargeaccumulated within photodiode 260 to be transferred to floatingdiffusion 262 and sensed by the pixel readout circuitry. Morespecifically, floating diffusion 262 is coupled to the gate of sourcefollower 255 (an amplification and/or charge-to-voltage conversionelement), which is itself coupled between a supply rail (V_(DD) in thisexample) and a read-out line, Vout, to enable a signal representative ofthe floating diffusion potential to be output to read-out logic outsidethe pixel.

As shown, a row-select transistor 257 is coupled between the sourcefollower and the read-out line to enable multiplexed access to theread-out line by respective rows of pixels. That is, row-select lines(“RS”) are coupled to the control inputs of row-select transistors 257within respective rows of pixels and operated on a one-hot basis toselect one row of pixels for sense/read-out operations at a time. Areset transistor 259 is also provided within the progressive read-outpixel to enable the floating diffusion to be switchably coupled to thesupply rail (i.e., when a reset-gate line (RG) is activated) and thusreset. The photodiode itself may be reset along with the floatingdiffusion by fully switching on transfer gate 251 (e.g., by assertingTGc while TGr is high) and reset transistor 259 concurrently, or bysimply connecting the photodiode to a reset-state floating diffusion.

FIG. 11 is a timing diagram illustrating an exemplary pixel cycle withinthe progressive read-out pixel of FIG. 10. As shown, the pixel cycle issplit into five intervals or phases corresponding to distinct operationscarried out to yield an eventual progressive read-out in the final twophases. In the first phase (phase 1), a reset operation is executedwithin the photodiode and floating diffusion by concurrently assertinglogic high signals on the TGr, TGc and RG lines to switch ontransfer-enable transistor 253, transfer gate 251 and reset transistor259, thereby switchably coupling photodiode 260 to the supply rail viatransfer gate 251, floating diffusion 262 and reset transistor 259 (theillustrated sequence can begin with an unconditional reset, e.g., at thestart of a frame, and can also begin from a preceding conditionalread-out/reset operation). To conclude the reset operation, the TGr andRG signals (i.e., signals applied on like-named signal lines) arelowered, thereby switching off transfer gate 251 (and sense gate andreset transistor) so that the photodiode is enabled to accumulate (orintegrate) charge in response to incident light in the ensuingintegration phase (phase 2). Lastly, although the row-select signal goeshigh during the reset operation shown in FIG. 11, this is merely aconsequence of an implementation-specific row decoder that raises therow-select signal whenever a given row address is decoded in connectionwith a row-specific operation (e.g., raising the TGr and RG signalsduring reset directed to a given row). In an alternative embodiment, therow decoder may include logic to suppress assertion of the row-selectsignal during reset as indicated by the dashed RS pulse in FIG. 11.

At the conclusion of the integration phase, the floating diffusion isreset (i.e., by pulsing the RG signal to couple the floating diffusionto the supply rail) and then sampled by a sample-and-hold element withinthe column read-out circuit. The latter operation, in effect, samplesthe noise level of the floating diffusion and is executed in theembodiment shown by asserting the row-select signal for the pixel row ofinterest (i.e., the “i^(th)” pixel row, selected by RSi) while pulsing areset-state sample-and-hold signal (SHR) to convey the state of thefloating diffusion to the sample-and-hold element (e.g., aswitch-accessed capacitive element) within the column read-out circuitvia read-out line Vout.

After acquiring the noise sample in phase 3, an over-threshold detectionoperation is executed in phase 4 by raising the TGr line to apartially-on, “over-threshold-detection” potential, VTG_(partial),concurrently with switching on transfer-enable transistor 253 (i.e., byasserting a logic high TGc signal, although in this embodiment TGc isalready on). By this operation, illustrated graphically in FIGS. 12 and13, VTG_(partial) is applied to the control node of transfer gate 251 toswitch the transfer gate to a “partial on” state (“TG partial on”).Referring to FIGS. 12 and 13, electrostatic potential diagrams forphotodiode 260 (a pinned photodiode in this example), transfer gate 251and floating diffusion 262 are shown below their corresponding schematiccross-section diagrams. Note that the depicted levels of electrostaticpotential are not intended to be an accurate representation of thelevels produced in an actual or simulated device, but rather a general(or conceptual) representation to illustrate the operation of the pixelread-out phases. Upon application of VTG_(partial) to the control nodeof transfer gate 251, a relatively shallow channel potential 271 isformed between photodiode 260 and floating diffusion 262. In the exampleof FIG. 12, the level of charge accumulated within the photodiode at thetime of the over-threshold detection operation (phase 4) is insufficientto enable charge transfer via the shallow channel potential of thepartially-on transfer gate. Accordingly, because the accumulated chargelevel does not exceed the spillover threshold established by applicationof VTG_(partial) to the control node of transfer gate 251, there is nospillover from the photodiode to the floating diffusion and theaccumulated charge instead remains undisturbed within the photodiode. Bycontrast, in the example of FIG. 13, the higher level of accumulatedcharge does exceed the spillover threshold so that a portion of theaccumulated charge (i.e., that subset of charge carriers that are abovethe transfer gate partially-on electrostatic potential) spills over intofloating diffusion node 262, with the residual accumulated chargeremaining within the photodiode as shown at 272.

Still referring to FIGS. 11, 12 and 13, at the conclusion of theover-threshold phase, the charge level of the floating diffusion issampled and held within a signal-state sample-and-hold element (i.e., inresponse to assertion of signal SHS) to yield a threshold-testsample—the difference between the signal-state sample and the previouslyobtained reset-state sample—to be evaluated with respect to aconditional-reset threshold. In one embodiment, the conditional-resetthreshold is an analog threshold (to be compared with an analogthreshold test sample in response to a compare-strobe signal using asense amplifier for example) set or programmed to a setting above thesampling noise floor, but low enough to enable detection of minutecharge spillover via the shallow transfer gate channel. Alternatively,the threshold-test sample may be digitized in response to assertion of aconvert-strobe signal (e.g., within an analog-to-digital converter thatis also used to generate the finalized pixel sample value) and thencompared with a digital conditional-reset threshold, again, set (orprogrammed to a setting) above the noise floor, but low enough to enabledetection of trace charge spillover. In either case, if thethreshold-test sample indicates that no detectable spillover occurred(i.e., threshold-test sample value is less than conditional-reset spillcharge threshold), then the photodiode is deemed to be in theunder-threshold state shown in FIG. 12 and the TGc line is held low inthe ensuing conditional read-out phase (phase 5, the final phase) todisable transfer gate 251 for the remainder of the progressive read-outoperation—in effect, disabling further read-out from the photodiode andthus enabling the photodiode to continue integrating charge withoutdisruption for at least another sampling interval. By contrast, if thethreshold-testing sample indicates a spillover event (i.e.,threshold-test sample greater than conditional-reset spill chargethreshold), then the TGc line is pulsed on during the conditionalread-out phase concurrently with application of a fully-on,“remainder-transfer” potential, VTG_(full), to the TGr line, therebyenabling the remainder of the charge (272) within photodiode 260 to betransferred to floating diffusion 262 via the full-depth transfer-gatechannel (273) so that, between the over-threshold transfer in phase 4and the remainder transfer in phase 5, the charge accumulated within thephotodiode since the hard reset in phase 1 is fully transferred to thefloating diffusion where it may be sensed in a pixel read-out operation.In the embodiment shown, the pixel-readout operation is effected bypulsing the SHS signal and compare/convert strobe in sequence duringconditional read-out phase 5, though either or both of those pulses mayoptionally be suppressed in absence of an over-threshold detection. Notethat conditional read-out of the photodiode (i.e., effected by pulsingTGc in conjunction with application of VTG_(full) on TGr) effectivelyresets the photodiode, while suppression of the conditional read-outleaves the integration state of the photodiode undisturbed. Accordingly,execution of the conditional read-out operation conditionally resets thephotodiode in preparation for integration anew in the succeedingsampling interval (subframe) or refrains from resetting the photodiodeto enable cumulative integration in the subsequent sampling interval.Thus, in either case, a new integration phase follows phase 5, withphases 2-5 being repeated for each subframe of the overall frame (orexposure) interval, before repeating the hard reset in a new frame. Inother embodiments, where cumulative integration is permitted acrossframe boundaries, the hard reset operation may be executed to initializethe image sensor and omitted for an indeterminate period of timethereafter.

FIG. 14 illustrates an embodiment of an image sensor 300 having aprogressive-readout pixel array 301, sequencing logic 303, rowdecoder/driver 305 and column read-out circuit 307. While pixel array301 is shown to include four rows and two columns of shared-elementpixels, other embodiments may include many more pixel rows and columnsto implement, for example, a multi-megapixel or gigapixel image sensor.The column read-out circuit 307 (for which two columns of read-outcircuitry are depicted) and row decoder/driver 304 may likewise bescaled to meet the number of pixels in the pixel array.

In the embodiment shown, each column of the pixel array is populated byshared-element pixels in which every four pixels form a quad pixel cell310 and contain respective photodiodes 260 (PD1-PD4), transfer gates251, and transfer-enable gates 253, but share a floating diffusion node312, reset transistor 259, source follower 255 and row-select transistor257. By this arrangement, the average transistor count per pixel is 2.75(i.e., 11 transistors/4 pixels), thus effecting a relatively efficient,2.75T-pixel image sensor.

As shown, row decoder/driver 305 outputs a shared row-select signal (RS)and reset-gate signal (RG) to each row of quad-pixel cells 310, andoutputs independent row transfer-gate control signals (TGr1-TGr4) todrain terminals of respective transfer-enable transistors 253. In anembodiment in which row decoder/driver 305 sequences incrementallythrough the rows of the array (e.g., pipelining reset, integration andprogressive read-out operations with respect to the rows of pixel array301 such that one row is read-out after another), the row decoder/drivermay include logic to assert the RG, RS and TGr signals at theappropriate time for each row (e.g., synthesizing those signals withrespect to a row clock from sequencing logic 303). Alternatively, rowdecoder/driver 305 may receive individual timing signals correspondingto each or any of the RG, RS and TGr signals, multiplexing anyindividual enable pulse onto the corresponding RG, RS, or TGr lines of aselected row at the appropriate time. In one embodiment, the rowdecoder/driver receives transfer-gate control voltages corresponding tothe off, partially-on and fully-on states shown in FIGS. 11, 12 and 13(i.e., VTG_(off), VTG_(partial), VTG_(full)) from an on-chip or off-chipprogrammable voltage source 309, switchably coupling each of thedifferent control voltages to a given transfer-gate row line at adeterministic time, for example, as shown in FIG. 11. In alternativeembodiments, more than one voltage source 309 may be provided withinimage sensor 300 to enable transfer-gate control voltages to be locallycalibrated and thus compensate for control-voltage and/or performancevariations (i.e., non-uniformity) across the pixel array.

Still referring to the embodiment of FIG. 14, column read-out circuit307 includes a bank of read-out circuits 315, each implementing adigital threshold comparator and a relatively low bit-depthanalog-to-digital converter (e.g., a 4-10 bit ADC, though lower orhigher bit depth ADCs may be employed) to execute the over-thresholddetection and conditional sampling operations, respectively, discussedin connection with FIGS. 11-13. In one implementation, the thresholdcomparator and ADC are implemented by separate circuits so that thepixel sample value may be generated without regard to theconditional-reset threshold applied in the over-threshold determination.Through this approach, the conditional-reset threshold is dissociatedfrom the reference signals (“ADC Vrefs”) used in the ADC conversion,freeing the conditional-reset threshold and ADC reference voltages to beindependently adjusted (e.g., through reprogramming athreshold-reference generator) dynamically during or prior to sensoroperation to achieve calibration and/or compensate for changingoperating conditions or sub-optimal imaging results. In an alternativeembodiment, the threshold comparator may be implemented as part of theADC (e.g., using a reference applied in connection with resolving thedigital sample value as the conditional-reset threshold), potentiallyreducing the footprint of the column read-out logic through more compactcircuit design.

In the embodiment shown, the sequencing logic delivers a column clock,sample-and-hold strobes (SHR, SHS), and compare/convert strobe to thecolumn read-out logic to enable the operational timing shown, forexample, in FIG. 11. That is, during the over-threshold detection phase(i.e., phase 3), the read-out circuit for a given pixel column asserts(or maintains assertion of) the TGc line so that, when the rowdecoder/driver switches the TGr line for a given pixel row to thepartially-on potential (e.g., VTG_(partial), which is applied to thetransfer gates of the pixel row), execution of the over-thresholddetection operation described above is enabled. Accordingly, thethreshold comparator within each read-out circuit evaluates the state ofthe threshold-test sample (which is generated according to the state ofshared floating diffusion 312 following application of VTG_(partial) tothe transfer gate of a given photodiode) with respect to theconditional-reset threshold to yield a binary over-threshold result. Ifan over-threshold condition is detected, the read-out circuit raises theTGc signal again a short time later (i.e., in conjunction with thefully-on TGr potential (VTG_(full)) to effect a conditional read-outoperation, enabling a full read-out of the photodiode state onto Voutand resetting the photodiode) and executes an analog-to-digitalconversion operation in response to assertion of the compare/convertstrobe to yield a digitized pixel sample.

Correlated Double Sampling with Conditional Charge Restoration

FIG. 15 illustrates an alternative conditional-reset pixel embodiment330 having a transfer gate 333 disposed between a photosensitive element331 (e.g., a pinned photodiode) and a gate-controlled sense node 335 toenable correlated double-sampling. As shown, sense node 335 isestablished by a sense gate 337 (e.g., the channel underlying the sensegate forms the sense node in response to a pre-charge potential VPGapplied via photogate control transistor 341) and capacitively coupledto the gate of a source follower 339 via the gate terminal of the sensegate. Sense gate 337 is disposed between transfer gate 333 and a resetgate 343, which, as discussed below, enables charge transferred fromphotodiode 331 to sense node 335 to be either (i) discharged to a supplyvoltage node 340 (Vdd/Vrst) via reset gate 343 to effect a resetoperation, or (ii) transferred back onto the photodiode 331 via transfergate 333 to enable further charge integration, depending, for example,on whether the charge level read-out of the pixel exceeds a conditionalreset threshold. Moreover, disposition of the transfer gate betweensense node 335 and photodiode 331 enables a correlated double samplingoperation, sampling the sense node before and after charge-transfer fromthe photodiode to yield a high SNR pixel read-out via source follower339 and a row-select transistor 325. Thus, altogether, pixelarchitecture 330 enables a low-noise correlated double-sampling of thephotodiode charge state, followed by either a reset operation or acharge restoration operation (i.e., driving charge back onto thephotodiode) depending on whether the read-out result indicates anover-threshold condition within photodiode 331.

FIG. 16 illustrates exemplary operational phases within a pixel cycle ofthe conditional-reset pixel of FIG. 15, and FIG. 17 presents acorresponding timing diagram showing exemplary control signal statesgenerated during each phase of operation. FIGS. 18A-18G illustrateexemplary electrostatic potential states of the conditional-reset pixelduring the operational phases shown in FIGS. 16 and 17. Referring toFIGS. 16 and 17, a hard reset (361) is carried out in phase 1 of thepixel cycle by setting the precharge voltage source (VPG) to a logichigh level and asserting the row-select (RS), transfer gate (TG),photogate (PG) and reset gate (RG) signals (the last of which may begenerated by logically ANDing row and column control signals to enablereset at pixel granularity). By this operation, the transfer gate, sensegate and reset gate are switched on (the sense gate being switched on byvirtue of the switched coupling of the sense gate control node to VPGvia the photogate control transistor) to form a conductive channelbetween the photodiode and supply voltage rail (e.g., V_(DD) or Vrst).The state of the conditional-reset pixel during the reset operation isshown in FIG. 18A, with the conductive channel formed beneath thetransfer gate, source gate and reset gate shown at 381, 383 and 385,respectively.

To conclude the reset phase and prepare for charge integration (i.e.,charge accumulation in response to light incident upon the photodiode),the conductive channel between the photodiode and supply voltage rail(i.e., 381/383/385) is pinched off starting at the transfer gate andprogressing toward the supply rail node. That is, the transfer gate isswitched off first, then the sense gate, and then the reset gate, asshown in FIG. 18B, and by the successive falling edges of the TG, PG andRG control signals in FIG. 17, thus driving residual charge out of thecollapsing channel to the supply voltage rail.

After concluding the photodiode reset operation in phase 1, the transfergate, photogate control transistor and reset gate are maintained in thenon-conducting state throughout an integration phase 363 (phase 2) inwhich charge is integrated (accumulated) within the photodiode inresponse to incident light as illustrated by the electron population(“e”) in FIG. 18C.

Immediately following integration phase 363, a read-out phase 365 iscommenced by pulsing the photogate control signal (PG) to effectformation of a sense node (i.e., as shown at 391 (“SN”) in FIG. 18D) inpreparation for a signal sensing operation, and raising the row-selectsignal to couple the output of the source follower to the Vout line(i.e., bit line). Note that the bit line is pulled up to near Vdd bythis operation and is thus pre-charged in preparation for a read-outoperation. After pulsing the photogate control signal to form the sensenode and pre-charge the Vout line, a correlated double samplingoperation is performed by (i) capturing a reset-state sample within areset-state sample-and-hold element of the column read-out circuit bypulsing a reset-state sample-and-hold strobe (SHR) to store the state ofthe Vout line, (ii) pulsing the transfer gate signal (TG) to establish aconduction path between the photodiode and sense node and thereby effecta charge transfer from the photodiode to the sense node as shown in FIG.18E, and then (iii) pulsing a signal-state sample-and-hold strobe (SHS)to store a sample of the photodiode state within a signal-statesample-and-hold element of the column readout circuit, and finally (iv)pulsing a compare/convert signal to trigger an A/D conversion of thedifference between the signal-state and reset-state samples.

As indicated in FIG. 16, the correlated double sample result obtainedduring the read-out phase 365 is compared with a conditional-resetthreshold to determine whether to reset the photodiode or to enablefurther integration without photodiode reset, the alternate conditionaloperations executed in phase 4. More specifically, if the sample resultexceeds the conditional-reset threshold, then a conditional reset 367 isperformed in phase 4 of the pixel cycle by pulsing the transfer-gate,photogate control transistor and reset gate signals as in hard-resetphase 1. By contrast, if the sample result does not exceed theconditional-reset threshold, then no reset operation is performed (i.e.,RG signal is held low) and instead a conditional charge restorationoperation 369 is executed by lowering the VPG potential to a level thatrepels charge from the sense node back onto the photo diode (i.e.,pushes charge back onto the photodiode), thus restoring the photodiodeto its pre-charge-transfer state. This operation is shown in FIGS. 18F(sense-node eliminated by lowering VPG) and 18G (transfer gate switchedoff to force charge back onto photodiode). Accordingly, the photodiodestate is either reset to enable integration to commence anew or restoredto its charge-accumulated state to enable cumulative integration (i.e.,continue integrating from the photodiode state that existed prior to thephase 3 charge-transfer operation), depending on whether the photodiodestate sampled in read-out phase indicates an over-threshold condition.In either case, a further integration phase is commenced following phase4, with phases 2-4 being repeated according to the number of samplingintervals (or subframes) per frame interval.

FIG. 19 illustrates a more detailed embodiment of a conditional-resetpixel 410 capable of executing theconditional-reset/conditional-restoration operations described inreference to FIGS. 16-18G. In addition to the photodiode 331, transfergate 333, sense gate 337, source follower 339, photogate transistor 341,reset gate 343, and row-select transistor 345 described in reference toFIG. 15, conditional-reset pixel 410 includes a reset-enable transistor412 to effect a logic AND of a column-reset signal (RST) and row-selectsignals (RS), thus enabling assertion of the reset gate signal (RG)within selected (individual) pixels of a row, and a reset operation tobe performed at individual pixel granularity. In the implementationshown, a reset signal (RST) is provided vertically (i.e., from columnlogic) via a column base signal line and logically ANDed with therow-select device within switching element 412 to yield the reset gatesignal (RG) described in reference to FIG. 15. By this arrangement, hardreset and conditional reset operations within pixel 410 are effectedthrough concurrent assertion of both row and column control signals(row-select, RS and column-reset, RST) associated with a given pixel,and thus at pixel granularity (i.e., a single pixel within the pixelarray may be isolated for purposes of pixel reset operations).Accordingly, pixel 410 can be operated individually with respect to thepixel cycle phases described in reference to FIGS. 16 and 17, thusenabling a conditional reset for any individual pixel while alsoproviding for a correlated double sampling in a non-destructive readout.This proposed structure achieves an advantageous readout scheme for aconditional reset pixel sensor, enabling effective and accuratemeasurement of a given pixel in a significantly wider signal range thanobtained from conventional sensor readout. Like with the FIG. 14embodiment, FIG. 19 can also be arranged such that multiple photodiodes331 and transfer gates 333 share the other pixel circuitry of FIG. 19.

Analog Uncorrelated Double Sampling and Digitally Correlated DoubleSampling

FIG. 20 illustrates an embodiment of a conditional-reset 3T pixel 450and read-out circuitry 470 that permits sampling noise reduction throughboth digitally correlated double sampling and analog uncorrelated doublesampling. As shown, the 3T pixel includes a photodiode 451, a sourcefollower 453, a read-select transistor 455, and a reset-AND gate whichincludes two transistors 457 and 459 to perform a logic AND ofrow-select and column-reset signals and thus, when those two controlsignals are asserted, switchably couple the photodiode to the supplyrail. Read-out circuitry 470 includes a data-out line 471 (i.e., coupledto receive the photodiode state via read-select transistor 455),reference line 472 (i.e., coupled to an on-chip or off-chip referencevoltage source), sample-and-hold elements 473 and 475, ADC 477,selective complement logic 479 and memory 481.

In an analog uncorrelated double sample, following an integrationinterval, the photodiode state may be: captured within signal-statesample-and-hold element 473 (i.e., closing and then opening the switchwithin element 473 to sample and hold the charge level on capacitivenode S) in a signal sampling operation; reset through concurrentassertion of the column-reset (“col-reset”) and row-select signals; andthen sampled again to sample and hold the photodiode reset state withinreset/reference sample-and-hold element 475. The difference between thecharge-accumulated state and reset state of the photodiode (i.e.,captured within storage nodes S and R of elements 473 and 475,respectively) may then be digitized within ADC 477 to yield a finalsample value in which a systematic offset (i.e., a nonzero but repeatedportion of the reset state of the photodiode and/or source followeroffset) is canceled. The double sample thus obtained is referred toherein as an “uncorrelated” double sample because the photodiode resetoperation follows the signal sampling operation, meaning that anyresidual charge (e.g., thermal noise, kTC) on photodiode 451 followingthe reset operation occurs without correlation to any residual charge onthe photodiode in a previous reset that preceded the charge integrationinterval and signal sampling operations.

In the case of a digitally correlated double-sampling operation, thereset state of photodiode 451 (i.e., photodiode state immediately afterconcurrent assertion of row-select and col-reset signals) may besampled, digitized within ADC 470 and stored as a negative value formedwithin memory element 479 (i.e., conceptually passing throughcomplementing branch 480 of selective complement logic 479 on the way tomemory 481). After an integration interval transpires, thecharge-accumulated state of photodiode 451 may be sampled, digitized anddelivered without complement (i.e., via the non-complementing branch oflogic 479) to memory 481 and eventually summed with the negative valueof the reset-state sample to yield a correlated double sample. Theoverall sampling operation is referred to herein as a digitallycorrelated double-sampling due to the digital storage of the reset andcharge-accumulated states of photodiode 451. In digital correlateddouble sampling the sampling is performed by comparing both before andafter accumulation against the reference. In both cases the signal fromthe photo diode is stored on sample and hold capacitor 473, thereference is stored on sample and hold capacitor 475 and the ADCmeasures the difference between the samples stored in these two sampleand hold capacitors.

FIG. 21 is a flow diagram illustrating a combination of a digitallycorrelated double sampling operation with one or more analoguncorrelated double sampling operations that may be carried out toachieve a reduced-noise pixel read-out within the conditionally-reset 3Tpixel and read-out architecture of FIG. 20. At the start of an exposureor frame interval, a hard reset operation is performed at 501 to resetthe photodiode in preparation for photon-induced charge integration,followed at 503 by a sample of the photo-diode reset-state with respectto a reference signal (e.g., delivered via reference line 472 in FIG.2). The resulting “sample against reference” is stored as a negativevalue (i.e., passing through the complementing branch of logic 479)within memory at 505 (equivalently, the sample taken at 503 can bestored as a positive value, and then recalled later for subtractionagainst a second sample). After an integration period transpires (507),the photodiode state is again sampled with respect to the reference at509. If the immediately concluded integration period was the finalintegration period of a fixed frame interval (affirmative determinationat 511), then the sample acquired at 509 is added to memory at 521 toeffect a digitally correlated double sampling (i.e., establishing (orenabling determination of) the difference between the negativereset-state sample stored at 505 and the positive accumulated chargesample stored at 521). If the integration period at 507 is not the lastintegration period in the exposure interval (negative determination at511), then the sample acquired at 509 is compared with theconditional-reset threshold (θ) in decision operation 513. If the sampleindicates that the accumulated charge within the photodiode is less thanthe threshold (negative determination at 513), then the operations at507, 509 and 511 are repeated to permit charge accumulation to resumewithin the photodiode (i.e., no photodiode reset) through anotherintegration period, and followed again by a determination as to whetherthe final integration period has transpired and, if not, whether theconditional-reset threshold has been exceeded.

Still referring to FIG. 21, if the sample acquired at 509 indicates thatthe conditional-reset threshold has been exceeded (i.e., accumulatedphotodiode charge>θ), then a reset operation is executed at 515 to resetthe photodiode, followed by generation of a signal sample against thereset potential at 517. More specifically, the signal-state sampleremains within the signal-state sample-and-hold element (i.e., element473 of FIG. 20) following the immediately preceding sampling operationat 509, so that by switching the input source of the reset/referencesample-and-hold element (e.g., element 475 of FIG. 20) from thereference line to the Vout line and sampling the Vout line within thereset/reference sample-and-hold element, the previously acquiredsignal-state sample and newly acquired reset-state sample may bedelivered differentially (i.e., signal state minus reset state) to theanalog to digital converter to digitize the analog uncorrelated doublesampling. In effect, the determination that the pixel-state is to bereset (i.e., affirmative determination at 513), triggers a substitutionof the photodiode reset-state sample for the reference sample within thereset/reference sample-and-hold element, thus permitting generation ofan analog uncorrelated double sample.

As shown, the digitized value of the analog uncorrelated double sampleis added to the memory at 519 and then a new integration period andensuing operations are begun at 507. At the final integration interval,the sample against reference acquired at 509 is added to memory at 521as explained above, thus concluding the exposure interval. Accordingly,assuming that some non-zero number (N−1) of conditional reset operations515 are carried out within a given exposure interval, then theaccumulated sample set will be:[Ref₀−RS₀]+Σ_(i=1) ^(N-1)[SS_(i)−RS_(i)]+[SS_(N)−Ref_(N)]  (1),where “SS” is a signal-state (i.e., state of the charge-accumulatedphotodiode plus the prior reset state) sample, “RS” is a reset-statesample, “Ref” is a reference line state sample, and ‘−’ and ‘+’ denotesubtraction and addition, respectively. That is, N−1 analog uncorrelateddouble-sampling operations are bookended by a digitally correlateddouble-sampling operation. Moreover, because each analog uncorrelateddouble-sampling operation itself involves capture and differencing ofthe noise (e.g., kTC noise) associated with two different resetoperations (i.e., the one preceding the integration value sampled in anSS sample and the reset operation 515 following an intermediate SSsampling), the reset state captured with respect to one analoguncorrelated double sample is in fact correlated to the signal statecaptured with respect to a subsequent uncorrelated double sample. Thatis, expanding expression (1) above by defining SS(i)=RS(i−1)+CI(i),where CI is the charge integrated since the last reset operation plusquantization/other non-stationary noise, yields:[Ref₀−RS₀]+Σ_(i=1)^(N-1)[CI_(i)+RS_(i−1)−RS_(i)]+[CI_(N)+RS_(N-1)−Ref_(N)]  (2),which, upon re-associating, may be expressed as:Σ_(i=1) ^(N)CI_(i)+[Ref₀−Ref_(N)]  (3).Thus, the combination of analog uncorrelated double sampling operationsand the digitally correlated double-sampling operations yields, ineffect, a set of fully correlated double-sampling operations togetherwith a difference between the reference signal sample acquired at thestart and end of the integration interval, the latter being negligiblein the case of a relatively temporally noise-free reference source.

FIG. 22 illustrates a more detailed embodiment of the pixel architectureand read-out circuit of FIG. 20, showing exemplary switch settingswithin the signal-state and reset/reference sample-and-hold elementsduring (1) the sample against reference operations shown at 503 and 509in FIG. 21, and (2) the sample against reset operation shown at 517 inFIG. 21. As shown in FIG. 22, the read-out circuit, like all read-outcircuits disclosed herein, may include a gain element 501 to providegain greater than or substantially equal to unity. Also, two pixels 450(shown as 450 ₀ and 450 ₁) are illustrated instead of one, todemonstrate column line interconnections. FIG. 23 illustrates analternative embodiment of the pixel architecture presented in FIGS. 20and 22. As shown, reset transistors 553 and 557 are disposed in seriesbetween the photodiode and supply voltage node to form a logic AND gate.Also, a dedicated row-reset signal (“row_rst”) is provided to switchon/off transistor 557 and thus avoid the thermal noise injection thatmay otherwise occur if controlled by the row-select signal.

In conditional-reset image sensor embodiments discussed thus far,unconditional-reset operations are executed at the conclusion of eachnon-final subframe within an overall frame (or exposure) interval, withany residual pixel value (i.e., level of accumulated charge, whetherabove the conditional-reset threshold or not) being read out at theconclusion of the final subframe. Through this operation, referred toherein as “residue mode” read-out, a finalized pixel value may beconstructed for each frame using the non-final subframe read-outs andthe residue read-out (for instance, by summing and reference to a lookuptable).

Referring to the exemplary residue-mode read-out sequence for threedifferent charge accumulation patterns shown at 601 in FIG. 24, it canbe seen that, at extremely low-light intensities (i.e., intensity 1),the total charge accumulated in each of N frames never rises far abovethe noise floor, yielding relatively noisy dark pixels which areparticularly noticeable in the case of video frames. At the considerablybrighter intensity 2, the per-frame charge accumulation still fails toexceed the conditional-reset threshold (“Th”), but at least risessufficiently above the noise floor to yield a reasonable SNR. At theeven brighter intensity 3, the per-frame charge accumulation meets theconditional-reset threshold shortly before end-of-frame, but leaves alow-valued and thus low SNR end-of-frame residue that tends to degradethe overall SNR of the image frame.

FIG. 24 also illustrates, for purpose of contrast, application of analternative read-out mode, referred to herein as inter-frame integrationmode (IFI) or dynamic range extension mode (DRX), to the same threelight intensities. In the inter-frame integration mode, instead ofperforming an unconditional reset at the start of each frame for eachpixel, integrated charge remaining at the end of a given frame for agiven pixel is carried over to the next frame if the threshold is notexceeded on the last frame readout, effectively extending the maximumduration over which a pixel is enabled to continuously integrate charge.Through this operation, low-level light intensity may be integratedthrough a sequence of frames, rising well above the noise floor to yieldan eventual reset, or at least a meaningful residue read-out if theinter-frame integration is limited to a fixed number of frames. In thecase of an IFI read-out at intensity 1, for example, in contrast to thesequence of low SNR read-outs acquired in residue mode, chargeintegration continues through a sequence of N frames to an eventualreset event, thereby yielding a high SNR result. At intensity 2,inter-frame integration yields a steady sequence of reset events thatmay be used to generate a high SNR read-out, generally matching theperformance of residue mode. For higher intensity 3, the IFI approachenables the small level of residual charge at the end of each frame tocontribute to the charge integration in the next frame, thus avoidingthe degrading effect of the low-SNR residue read-outs.

Reflecting on FIG. 24, it can be seen that inter-frame integration, ineffect, eliminates the framing boundary as a demarcation point forfinalizing image pixel values, presenting a dilemma as to how to meetthe requirement for end-of-frame pixel value generation (e.g., in avideo imaging system which yields a steady output frame rate) in view ofincomplete inter-frame integration. Detail view 615 illustrates anapproach employed in a number of embodiments presented herein, includingrecording timestamps of reset events to enable determination of anelapsed “inter-frame integration interval” (IFI interval or IFI period)between the final reset event within a given frame N (the “in-framereset”) and the most recent reset from a prior-frame (the “pre-frame”reset), (ii) aggregating pixel sample values acquired at one or morereset events during the IFI interval to determine the total chargeintegrated within the pixel over the IFI interval, and then (iii)estimating or predicting an end-of-frame pixel value based on the IFIinterval and the total charge integrated over that interval. Estimatingthe end-of-frame pixel value is particularly challenging where one ormore successive image frames are entirely devoid of reset events andthus contain no non-zero samples for a given pixel (e.g., as in theexamples of intensities 1 or 2 in FIG. 24). In a number of embodiments,described in further detail below, the pixel value in such frames isestimated on the basis of the pixel value determined at the most recentreset event, and potentially attenuating the estimate as thelast-determined pixel value ages.

FIG. 25 illustrates an exemplary per-pixel frame processing approachthat may be employed within a still or video imaging system, leveragingthe inter-frame integration approach shown in FIG. 24 to yieldrelatively high-SNR images in low light conditions. As shown, twohigh-level operations are performed: first, at 625, an updated referencepixel value is generated, and corresponding reference timestamprecorded, in response to the final reset event occurring within thesubject image frame, if one or more in-frame reset events in factoccurred. Then, at 627, an output pixel value is estimated for the imageframe based on the reference pixel value and reference timestamp.

Reference pixel generation and timestamping within operation 625 may beimplemented by the component operations shown at 631 and 633. Morespecifically, as shown at 631, for each pixel exhibiting at least onenon-zero readout during the current frame (i.e., indicating that aconditional reset was executed), the imaging system determines, as theIFI interval, the elapsed time between the final in-frame reset and themost recent pre-frame reset, and also sums all non-zero read-out valuesacquired during the current frame to produce, as the IFI result (or IFIvalue), a value of the net amount of charge integrated within the pixelover the IFI interval. At 633, a reference pixel value is determinedbased on the duration of the IFI interval and magnitude of the IFIresult (e.g., through table lookup and/or calculation or heuristic), andthe timestamp of the final in-frame reset (i.e., the most recent resetevent and thus the end of the IFI interval) is recorded as the referencepixel time stamp. Continuing to component operation 635, whichcorresponds to high-level operation 627, an estimated pixel value isgenerated based on the reference pixel value and the elapsed time sincethe most recent reset event (i.e., indicated by the reference pixeltimestamp), attenuating the current-frame pixel estimate relative to thereference pixel value (and updating the reference pixel value to reflectthe new estimate) where the reference pixel timestamp indicates that noIFI result was obtained in the current frame and the elapsed timeindicates that the light intensity indicated by the prior-frame estimatehas dropped. This latter circumstance, referred to herein as “coastingpixel attenuation” is described in further detail below in reference toFIG. 28.

FIG. 26 illustrates an embodiment of an imaging system 650 capable ofgenerating image frames using the inter-frame integration approachoutlined in FIGS. 24 and 25. As shown, imaging system 650 includes animage sensor IC 651 (“imager”), image signal processor IC 653 (“ISP”)and memory IC 655. Though depicted as discrete integrated circuitcomponents, in alternative embodiments, the functions performed by oneor more of the ICs may be merged into another of the ICs and/or the ICsmay be interconnected components in a variety of different multi-chippackages, including a three dimensional IC stack (3D IC) with, forexample, the die of imager 651 being ground (or lapped or otherwisethinned) to enable backside illumination and having contact-sideinterconnection to elements within the ISP 653 and/or memory 655.

In the embodiment shown, imager 651 outputs raw frame data to ISP 653,at least conceptually outputting one subframe of read-out data afteranother (sf₀-sf_(m-1)) and, within each subframe, outputting out one rowof read-out data after another (r₀-r_(n-1)) (in fact, rolling shutterconstraints and/or different schedules for different rows may precludesuch an orderly approach). In one embodiment, imager 651 and ISP 653each assume a predetermined subframe duration and row read-out order sothat the imager need not provide subframe or row identifiers within (orin association with) the raw data output. In alternative embodiments,imager 651 may tag the raw frame data as necessary with row identifiersand time stamps to explicate the frame data organization. In eithercase, ISP 653 stores each incoming raw frame data within a raw framedata buffer 661 (in memory 655), e.g., until the data can be processedto create output frame values. In one embodiment, ISP 653 operates onthe subframes and subframe rows of the raw frame data within buffer 661concurrently with receiving and loading new raw frame data into buffer661, pipelining new output frame generation (“frame out”) with receiptof raw frame data from imager 651, for example, at the frame rateestablished by the imager. In an alternative embodiment, ISP 653 mayoperate on all or part of the incoming raw frame data as it is received,buffering intermediate data (i.e., partially processed data) asnecessary to produce a finalized output frame “frame out.”

For purposes of explanation only, each pixel value output by imager 651as part of the raw frame data is assumed to include eight 12-bit pixelsamples generated, respectively, by sixteen subframe readouts (i.e., 16×temporal oversampling and 10-bit ADC within imager 651, with some of thesubframe outputs pre-summed by the imager prior to raw subframetransmission as explained below), so that raw frame data buffer 661 issized to permit storage of 96 bits per pixel (bpp). A larger or smallerraw frame data buffer may be implemented (or set through production-timeor run-time mode-register programming) to accommodate different numbersof subframes per frame and/or different bit-depth ADC implementations orconfigurations.

In an IFI operating mode in which there is no final-frame residuereadout (e.g., a video imaging IFI mode), only those pixel samplesacquired in connection with over-threshold events (and thus pixel reset)will be non-zero values. Accordingly, at least in low-light conditions,a substantial number of the pixel samples will be zero-valued, enablingsubstantial compression in chip-to-chip data transmission (in someexamples, intermediate subframe compression can produce on the order ofone bit-per-pixel output). By contrast, in a residue operating mode asshown at 601 of FIG. 24, a residue read-out is executed at theconclusion of the final subframe of each frame, ensuring in the presentexample that at least 12% (⅛) of the raw frame data will likely benonzero and not easily compressible, thus requiring higher averagechip-to-chip signaling bandwidth and memory bandwidth within imagingsystem 650. Note that a final-frame residue readout may be executed inconnection with a still-imaging IFI mode. For example, a finite number(N−1) of raw frames may include IFI data only (no residue readout)which, after being processed as described below, are algorithmicallycombined with a final (N^(th)) raw frame containing a final-subframeresidue readout. The value of N may be set by an imaging system operator(i.e., via a user-interface coupled to ISP 653 or another IC withinimaging system 650) according to conditions at hand, thus enablingcontinuous low-light accumulation throughout the N frames, concluded bya residue readout to finalize the still output image.

In addition to loading data from imager 651 into raw frame data buffer661, ISP 653 tracks last time stamps of reset events for each pixelwithin a reset timestamp buffer 663, and also maintains the estimatedpixel values of the most recently output frame within last-frame buffer655 (e.g., resolved to 12 bpp in the example shown, though higher orlower bit output pixel resolutions may be generated and recorded inalternative implementations), either of which may be implemented withinmemory 655 as shown, or in a separate memory element. As mentionedbriefly above and explained in further detail below, the timestamps areused to determine IFI intervals and the “last frame” values are appliedin pixel value estimations for coasting pixels (i.e., pixels for whichno reference pixel value is determined within the frame beingprocessed). In one embodiment, explained in further detail in connectionwith FIG. 27, a three-bit timestamp code is used to distinguish eightactual or approximate end-of-subframe times within each frame, and afive-bit frame number is used to identify prior frames, thus yielding an8 bpp time stamp that resolves to the approximate subframe endtimeswithin the most recent 32 frames. Larger timestamps (i.e., more bpp) maybe recorded in alternative embodiments to enable a larger frame-historywindow and/or more temporal resolution per frame, or the timestamp mayuse a quasi-floating-point format where coarser resolution and largerrange are used to record timestamps older than 32 frames.

FIG. 27 illustrates an exemplary subframe organization and timecodeassignment that may employed within the imaging system of FIG. 26. Asshown, individual subframes (of which only the longer subframes, sf₀,sf₈ and sf₁₅ are explicitly labeled) are organized so that samplingevents are clustered in time (e.g., longer subframe sf₀, followed byfour relatively brief subframes sf₁-sf₄; longer subframe sf₈ followed byfour relatively brief subframes sf₉-sf₁₂), thus enabling an approximateendtime of two groups of five subframes (sf₀-sf₄ and sf₈-sf₁₂) to berepresented by single respective timestamp codes. In such an embodiment,the imager may allocate a temporary buffer to, for each pixel on a givenrow, sum the results of sf₀-sf₄ prior to transmission to the ISP, andagain sum the results of sf₈-sf₁₂ prior to transmission (during thetimes the buffer is not needed for this row it can be allocated tocreating similar sums for other rows). Due to the summing operation theISP may not be able to know which one or more of the five summed samplesactually exceeded the threshold for a given pixel. But because these twogroups of subframes are gathered, respectively, over only 2.6% and 3.6%of the frame time, temporal resolution (which has an uncertainty of onlyhalf these intervals) is not greatly impacted. At the same time,however, summing allows a reduction by half of the number of subframesof raw data passed to the ISP, at the expense of provisioning, on theimager, shared summing buffers approximately 1/16^(th) the size of theimage array. By this design, the number of bits needed to resolve thesubframe endtimes is reduced from four to three—in effect, doubling thesize of the frame history window for a given timestamp bit depth. Inalternative embodiments, e.g., those without imager aggregation ofmultiple subframes, subframe endtimes may be fully resolved by thesubframe timecode (e.g., log₂ N bits allocated to subframe time code toresolve N subframes). Also, while non-uniform subframe intervals aredepicted (i.e., non-uniform sampling intervals), uniform samplingintervals may be used in alternative embodiments. Further, while thesame subframe intervals (whether uniform or not) are assumed for all therows if image pixels in the examples that follow, different subframeinterval progressions may be employed and/or the interval progressionsmay be staggered from row to row. In such embodiments, each row may betagged with a relative or absolute timestamp related to the time ofpixel readout to allow the ISP to calculate exposure intervals.

Having a frame defined with relatively long subframes at the beginningand ends of the frame has several advantages. One is shown in theoperation described above—a long subframe followed by several shortsubframes may allow all to be summed efficiently on the imager. Second,placing short subframes more central to the frame instead of at one endmay reduce motion artifacts caused by a moving object with differenttones having those tones resolved at far distant portions of the frame.Third, placing a long subframe before a short one virtually guaranteesthat pixels receiving significant illumination will be reset at thebeginning of each short sequence, thus increasing predictability andperformance for highlight tones. Fourth, placing a long subframe at theend of the frame virtually guarantees that highlight tones will exceedthe threshold and reset at frame end. Since highlight tones may exist ina non-linear region of the imager response curve, it is preferable thatthese tones be calculated on an integer-frame basis to avoid additionalcomplexity.

FIG. 28 illustrates an exemplary approach to estimating the output framevalues for pixels which yield no non-zero sample values (i.e., no resetevents) and thus are “coasting” during a given frame. In the embodimentshown, the reference pixel value is applied as an initial estimate for acoasting pixel, but may be attenuated as the reference pixel value ages.By this approach, an initially high-intensity pixel value (indicating abright point) for which over-threshold events suddenly cease (i.e.,scene elements and/or the imaging platform are moving, or a light sourcehas moved or been switched off) will be progressively dimmed in theoutput frame sequence. In one embodiment, for example, a filteringoperation is performed by determining the maximum end-of-frame pixelvalue that could have been reached by a coasting pixel (i.e., on theassumption that the pixel was on the cusp of an over-threshold event atframe-end), and applying that theoretical maximum, or “coast value,” ina filter with the reference pixel value (i.e., a value from thelast-frame buffer as the reference pixel value is updated from frame toframe). For example, an infinite impulse response filter is employed inone embodiment, though a finite impulse response filter (FIR) or anyother filtering technique or heuristic may be used). Referring to theestimation profile shown in FIG. 28 for a previously high-intensity, butnow coasting pixel, it can be seen that, after coasting for a firstframe (plus possibly a residual portion of the immediately prior,reference-pixel frame), the estimated intensity drops in accordance witha coast value equal to the conditional-reset threshold divided by theelapsed time in frames. After coasting for a second frame, the coastvalue and thus the filtered pixel estimation drops to a lower value(i.e., two frames have now passed without an over-threshold event, sothat the theoretical maximum rate at which photos are being accumulatedwithin the pixel is accordingly lower), and so forth for subsequentframes. As can be seen, the net effect is a frame-by-frame attenuationin the estimated pixel value of the coasting pixel. Note that the sameeffect occurs for lower intensity pixels, except that the coast value isnot applied to filter the estimate until the coast value drops below thepixel value estimated for the preceding frame (i.e., the most recentlyupdated reference pixel value).

FIG. 29 illustrates an exemplary frame processing sequence 700 that maybe executed by the ISP of FIG. 26 to implement the inter-frameintegration techniques described in reference to FIGS. 24-28. Ingeneral, the ISP generates one row of output frame data at a time andthus processes the row, data from all subframes (i.e., obtained from theraw frame data buffer), before proceeding to process the data forrow_(i+1). Accordingly, the ISP executes an inner “subframe loop” bysequencing through all subframes with respect to a given row, beforeadvancing row to row in an outer “row loop.” Though not specificallyshown, an additional “column loop” is executed within the subframe loop,to process each pixel within a given row. This loop is implied in FIG.29 by a column index, “col” which is sequenced from 0 to N−1 (N beingthe number of pixel columns in the imager) within each row-processingoperation.

Still referring to FIG. 29, at the start of a new frame, a row index(“row”), subframe index (“SF”), in-frame time stamp (“FTS”) and IFIvalue (“IFI”) are initialized to zero as shown at 701. Thereafter, at703, a row of raw image data specified by the subframe and row indicesis read from the raw frame buffer. If the raw image data is nonzero fora given pixel (i.e., negative determination at 705, as detected withinthe column loop), the in-frame time stamp is updated to reflect thetimecode corresponding the subframe index, and the raw image data (i.e.,pixel value) is accumulated within the IFI value, both as shown at 707.The timestamping and IFI value accumulation are skipped in the case ofzero-valued data (affirmative determination at 705). The subframe indexis evaluated at 709 to determine whether all subframes have beenprocessed for the current row index. If not, the subframe index isincremented and the operations at 703, 705 and 707 repeated.

After all the subframes corresponding to a given pixel row have beenprocessed, the ISP evaluates the IFI value to determine whether anyover-threshold events have occurred during the frame under process. Ifone or more over-threshold events have occurred (i.e., nonzero IFI valueand thus a negative determination at 715), the IFI interval isdetermined at 717 based on the elapsed time between the in-frame timestamp and reference timestamp. In the embodiment shown, a “time( )”function is invoked to yield a frame-referenced timestamp for the finalin-frame reset event, for example, based on a tuple of the current framenumber and subframe timecode, though other techniques may be used. At719, the IFI values and IFI intervals for the current row of pixels(i.e., at least those having non-zero IFI values) are applied inrespective image pixel calculations (which may be achieved, at least inpart, through a table lookup operation as indicated by the “Imap( )”function), with each resultant image pixel value being assigned as theestimated output image pixel value and also recorded as the newreference pixel value. At 721, a frame-referenced in-frame time stamp isassigned as the new reference pixel time stamp.

Returning to decision 715, for pixels that are coasting in the currentframe (i.e., having a zero IFI value) an affirmative determinationresult at decision 715, followed by a determination of the coast time at725 (elapsed interval between the end of the current frame, returned byTime(FTS[SF]),” and the reference pixel time stamp), and a determinationof the coast value at 727 (conditional-reset threshold divided by thecoast time). If the coast value is greater than the reference pixelvalue (i.e., affirmative determination at 729), then the reference pixelvalue is assigned to be the estimated output pixel value for the currentframe at 733. Otherwise, the estimated output pixel value is determinedbased on the coast value at 731, for example in a filtering operationthat blends the coast value and reference pixel value, or by simplyassigning the coast value to be the estimated output pixel value.

After the estimated output pixel value and any reference pixelvalue/timestamp updates have been recorded for the current row, the rowindex is incremented at 735, and the subframe, in-frame time stamp andIFI values reset in preparation for processing the next row of raw framedata.

FIG. 30 contrasts the dynamic ranges and SNR achievable for a given setof imager parameters, with and without the inter-frame integrationdescribed above. FIG. 30 also contrasts the effective frame rates withand without IFI. As shown, the IFI approach exhibits substantiallyimproved dynamic range, extending the low light sensitivity (maintaininga 15 db SNR) in this example by approximately four F-stops whileconverging to the dynamic range profile of residue mode operation athigher light intensities. While the frame rate is steady in residuereadout (new data is acquired for each pixel every frame at a 60 Hzframe rate in the example shown), the effective frame rate in the IFIapproach drops with light intensity as the elapsed time between pixelreset events grows. The rolloff in frame rate and asymptotic SNR floorare determined by the threshold number of charge carriers required tocause a readout event of a conditional reset pixel. For the exampleshown, with a threshold of 60, the effective frame rate for a pixel ishalved when the average converted photon arrival rate is 30 per nominalframe. A higher threshold would shift these numbers (e.g., a thresholdof 256 would cause the SNR to flatten out at about 24 dB, and half theframe rate at an average converted photon arrival rate of 128 (where theresidue mode would produce an SNR of approximately 20 dB).

In a given embodiment, various mechanisms can be used to further tailorthe response curve. For instance, the system can force a residue readoutat a given fraction of the base frame rate, such as every six frames, tolimit reduction in frame rate for a given pixel. The system can also usea “softer” encouragement without forcing a residue readout, such as byusing two thresholds. For instance, in the FIG. 27 sequence, a thresholdof 1024 photons can be used for subframes 0 through 14, and a thresholdof 64 photons can be used for subframe 15, such that much higher SNRreadings are obtained for all but the last subframe of a frame. Forreduced frame rate pixels, transitions to a new output value can also besmoothed by temporal combination of the old and new pixel values at thetransition frame.

Although the preceding description is laid out for a video-modeoperation, the same principles can be applied for still frame capture.For instance, a user can select a low-light-enhanced still frame mode,where a base shutter speed defines a base exposure time, which isaccompanied by an extended exposure time. As an example, a base shutterspeed of 1/60^(th) of a second results in a 16.7 ms base exposure timewith, e.g., 16 subframes, and a 50 ms extended exposure time, with,e.g., 5 additional subframes. For low-light pixels that have notproduced an above-threshold reading by the end of the base exposuretime, those pixels are allowed to continue to integrate for up until theend of the extended exposure time (at which time a residue reading isproduced for all pixels). The ISP constructs the image based, e.g., onthe base exposure time for pixels that exceeded the threshold at leastonce during the base time, and for the first exceedance for pixels thatfirst exceeded the threshold during the extended exposure time, and forthe total exposure time for pixels that only produced a residue readout.

Additional Considerations

It should be noted that the various circuits disclosed herein can bedescribed using computer aided design tools and expressed (orrepresented), as data and/or instructions embodied in variouscomputer-readable media, in terms of their behavioral, registertransfer, logic component, transistor, layout geometries, and/or othercharacteristics. Formats of files and other objects in which suchcircuit expressions can be implemented include, but are not limited to,formats supporting behavioral languages such as C, Verilog, and VHDL,formats supporting register level description languages like RTL, andformats supporting geometry description languages such as GDSII, GDSIII,GDSIV, CIF, MEBES and any other suitable formats and languages.Computer-readable media in which such formatted data and/or instructionscan be embodied include, but are not limited to, computer storage mediain various forms (e.g., optical, magnetic or semiconductor storagemedia, whether independently distributed in that manner, or stored “insitu” in an operating system).

When received within a computer system via one or more computer-readablemedia, such data and/or instruction-based expressions of the abovedescribed circuits can be processed by a processing entity (e.g., one ormore processors) within the computer system in conjunction withexecution of one or more other computer programs including, withoutlimitation, net-list generation programs, place and route programs andthe like, to generate a representation or image of a physicalmanifestation of such circuits. Such representation or image canthereafter be used in device fabrication, for example, by enablinggeneration of one or more masks that are used to form various componentsof the circuits in a device fabrication process.

In the foregoing description and in the accompanying drawings, specificterminology and drawing symbols have been set forth to provide athorough understanding of the disclosed embodiments. In some instances,the terminology and symbols may imply specific details that are notrequired to practice those embodiments. For example, any of the specificnumbers of bits, signal path widths, signaling or operating frequencies,component circuits or devices and the like can be different from thosedescribed above in alternative embodiments. Additionally, links or otherinterconnection between integrated circuit devices or internal circuitelements or blocks may be shown as buses or as single signal lines. Eachof the buses can alternatively be a single signal line, and each of thesingle signal lines can alternatively be buses. Signals and signalinglinks, however shown or described, can be single-ended or differential.A signal driving circuit is said to “output” a signal to a signalreceiving circuit when the signal driving circuit asserts (orde-asserts, if explicitly stated or indicated by context) the signal ona signal line coupled between the signal driving and signal receivingcircuits. The term “coupled” is used herein to express a directconnection as well as a connection through one or more interveningcircuits or structures. Integrated circuit device “programming” caninclude, for example and without limitation, loading a control valueinto a register or other storage circuit within the integrated circuitdevice in response to a host instruction (and thus controlling anoperational aspect of the device and/or establishing a deviceconfiguration) or through a one-time programming operation (e.g.,blowing fuses within a configuration circuit during device production),and/or connecting one or more selected pins or other contact structuresof the device to reference voltage lines (also referred to as strapping)to establish a particular device configuration or operation aspect ofthe device. The term “light” as used to apply to radiation is notlimited to visible light, and when used to describe sensor function isintended to apply to the wavelength band or bands to which a particularpixel construction (including any corresponding filters) is sensitive.The terms “exemplary” and “embodiment” are used to express an example,not a preference or requirement. Also, the terms “may” and “can” areused interchangeably to denote optional (permissible) subject matter.The absence of either term should not be construed as meaning that agiven feature or technique is required.

The section headings in the above detailed description have beenprovided for convenience of reference only and in no way define, limit,construe or describe the scope or extent of the corresponding sectionsor any of the embodiments presented herein. Also, various modificationsand changes can be made to the embodiments presented herein withoutdeparting from the broader spirit and scope of the disclosure. Forexample, features or aspects of any of the embodiments can be applied,at least where practicable, in combination with any other of theembodiments or in place of counterpart features or aspects thereof.Accordingly, the specification and drawings are to be regarded in anillustrative rather than a restrictive sense.

What is claimed is:
 1. An integrated-circuit image sensor comprising: aphotosensitive element to accumulate charge in response to incidentlight; and a read-out circuit to: determine whether the chargeaccumulated within the photosensitive element exceeds a first threshold;convert an analog readout signal representative of the chargeaccumulated into a multi-bit digital value having a numeric value inproportion to a magnitude of the analog readout signal if the chargeaccumulated is determined to exceed the first threshold; reset thephotosensitive element to a nominal discharged state in preparation forfurther charge accumulation if the charge accumulated is determined toexceed the first threshold; and refrain from generating the multi-bitdigital value and from resetting the photosensitive element to thenominal discharged state if the charge accumulated is determined not toexceed the first threshold.
 2. The integrated-circuit image sensor ofclaim 1 wherein the read-out circuit comprises: a read-out node; and acontrol circuit to (i) enable charge in excess of the first threshold tobe transferred from the photosensitive element to the read-out node,(ii) sense a charge level of the read-out node after enabling thetransfer of charge in excess of the first threshold, (iii) determine,based on the sensed charge level of the read-out node, whether thecharge accumulated within the photosensitive element exceeds the firstthreshold, (iv) enable charge in excess of the nominal discharged stateto be transferred from the photosensitive element to the read-out nodein response to determining that the charge accumulated within thephotosensitive element exceeds the first threshold, and (v) generate theanalog readout signal according to a charge level of the read-out nodeafter enabling the charge in excess of the nominal discharged state tobe transferred to the read-out node.
 3. The integrated-circuit imagesensor of claim 2 wherein the control circuit comprises: acharge-transfer switching element coupled between the photosensitiveelement and the read-out node; and switch control circuitry to apply afirst control signal to the charge-transfer switching element to enablethe charge in excess of the first threshold to be transferred from thephotosensitive element to the read-out node and to apply a secondcontrol signal to the charge-transfer switching element to enable thecharge in excess of the nominal discharged state to be transferred fromthe photosensitive element to the read-out node.
 4. Theintegrated-circuit image sensor of claim 3 wherein the charge-transferswitching element comprises a control terminal coupled to receive thefirst and second control signals at respective times and that effects acharge-conducting channel between the photosensitive element and theread-out node having a first electrostatic potential in response to thefirst control signal and a second electrostatic potential in response tothe second control signal, the second electrostatic potential beinggreater than the first electrostatic potential.
 5. Theintegrated-circuit image sensor of claim 2 wherein the read-out circuitcomprises an analog-to-digital converter to generate, as the multi-bitdigital value, a digital representation of the analog signal.
 6. Theintegrated-circuit image sensor of claim 5, and wherein the controlcircuit comprises reset circuitry to switchably couple thephotosensitive element to a reset voltage node after the control circuitdetermines that the charge accumulated within the photosensitive elementexceeds the first threshold and the charge level of the read-out nodehas been sensed to enable generation of the multi-bit digital value. 7.The integrated-circuit image sensor of claim 1, wherein the readoutcircuit comprises: circuitry to (i) transfer the accumulated charge fromthe photosensitive element to a sense node to enable generation of theanalog readout signal representative of the accumulated charge and to(ii) transfer the accumulated charge back to the photosensitive elementin response to an indication that the analog readout signal does notexceed the first threshold.
 8. The integrated-circuit image sensor ofclaim 7 wherein the read-out circuit includes a first switching elementto switchably couple the sense node to a voltage source node to resetthe sense node in response to an indication that the analog readoutsignal exceeds the first threshold.
 9. The integrated-circuit imagesensor of claim 7 wherein the read-out circuit comprises a firsttransistor disposed between the photosensitive element and the sensenode and control circuitry to (i) assert a first pulse on a controlterminal of the first transistor to transfer the accumulated charge fromthe photosensitive element to the sense node, and (ii) assert a secondpulse on the control terminal of the first transistor to transfer theaccumulated charge back to the photosensitive element.
 10. Theintegrated-circuit image sensor of claim 7 wherein the read-out circuitcomprises an amplifier capacitively coupled to the sense node, theamplifier to generate, as the analog readout signal representative ofthe accumulated charge, an amplified representation of a signal,received from the sense node via the capacitive coupling, correspondingto the accumulated charge transferred to the sense node.
 11. A method ofoperation within an integrated-circuit image sensor, the methodcomprising: accumulating charge within a photosensitive element inresponse to incident light; and determining whether the chargeaccumulated within the photosensitive element exceeds a first threshold;converting an analog readout signal representative of the chargeaccumulated into a multi-bit digital value having a numeric value inproportion to a magnitude of the analog readout signal if the chargeaccumulated is determined to exceed the first threshold; resetting thephotosensitive element to a nominal discharged state in preparation forfurther charge accumulation if the charge accumulated is determined toexceed the first threshold; and refraining from generating the multi-hitdigital value and from resetting the photosensitive element to thenominal discharged state if the charge accumulated is determined not toexceed the first threshold.
 12. The method of claim 11 whereindetermining whether charge accumulated within the photosensitive elementexceeds a first threshold comprises: enabling charge in excess of thefirst threshold to be transferred from the photosensitive element to aread-out node; sensing a charge level of the read-out node afterenabling the transfer of charge in excess of the first threshold; anddetermining, based on the sensed charge level of the read-out node,whether the charge accumulated within the photosensitive element exceedsthe first threshold.
 13. The method of claim 12 wherein converting theanalog readout signal representative of the charge accumulated into amulti-bit digital value if the charge accumulated is determined toexceeds the first threshold comprises enabling charge in excess of thenominal discharged state to be transferred from the photosensitiveelement to the read-out node.
 14. The method of claim 13 whereinconverting the analog readout signal representative of the chargeaccumulated into the multi-bit di vital value if the charge accumulatedis determined to exceed the first threshold further comprises, afterenabling charge in excess of the nominal discharged state to betransferred to the read-out node, outputting, as the analog readoutsignal representative of the charge accumulated, a signal correspondingto the charge level of the read-out node to an analog-to-digitalconverter.
 15. The method of claim 14 wherein resetting thephotosensitive element to a nominal discharged state in preparation forfurther charge accumulation if the charge accumulated is determined toexceed the first threshold comprises switchably coupling thephotosensitive element to a reset voltage node after outputting thesignal corresponding to the charge level of the read-out node to theanalog-to-digital converter.
 16. The method of claim 13 wherein enablingcharge in excess of the first threshold to be transferred from thephotosensitive element to the read-out node comprises applying a firstcontrol signal to a charge-transfer switching element coupled betweenthe photosensitive element and the read-out node, and wherein enablingcharge in excess of the nominal discharged state to be transferred fromthe photosensitive element to the read-out node comprises applying asecond control signal to the charge-transfer switching element to enablethe charge in excess of the nominal discharged state to be transferredfrom the photosensitive element to the read-out node.
 17. The method ofclaim 16 wherein applying the first and second control signals to thecharge-transfer switching element comprises applying the first andsecond control signals to a control input of the charge-transferswitching element at respective times to effect, between thephotosensitive element and the read-out node, a charge-conductingchannel having a first electrostatic potential and a secondelectrostatic potential, respectively, the second electrostaticpotential being greater than the first electrostatic potential.
 18. Themethod of claim 11 wherein determining whether the charge accumulatedwithin the photosensitive element exceeds a first threshold comprisestransferring the accumulated charge from the photosensitive element to asense node to enable generation of the analog readout signalrepresentative of the accumulated charge and determining whether theanalog readout signal exceeds the first threshold, the method furthercomprising transferring the accumulated charge back to thephotosensitive element in response to a determination that the analogreadout signal does not exceed the first threshold.
 19. The method ofclaim 18 wherein transferring the accumulated charge from thephotosensitive element to the sense node comprises generating a firstpulse on a signal line coupled to a control terminal of a firsttransistor disposed between the photosensitive element and the sensenode, and wherein transferring the accumulated charge back to thephotosensitive element comprises generating a second pulse at on thesignal line coupled to the control terminal of the first transistor. 20.An integrated-circuit image sensor comprising: a photosensitive elementto accumulate charge in response to incident light; and means for:determining whether the charge accumulated within the photosensitiveelement exceeds a first threshold; converting an analog readout signalrepresentative of the charge accumulated into a multi-bit digital valuehaving a numeric value in proportion to a magnitude of the analogreadout signal if the charge accumulated is determined to exceed thefirst threshold; resetting the photosensitive element to a nominaldischarged state in preparation for further charge accumulation if thecharge accumulated is determined to exceed the first threshold; andrefraining from generating the multi-bit digital value and fromresetting the photosensitive element to the nominal discharged state ifthe charge accumulated is determined not to exceed the first threshold.